ocr-projects-archive / ocr-tech /docs /PAPER_RESULTS.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
54.3 kB
# Paper Results
This file is the shortest path from repo artifacts to a paper draft.
Use [PAPER_NOTES.md](PAPER_NOTES.md) for claim boundaries and framing.
Use this file for the concrete result tables and the branch chronology.
## Current Comparison Tables
## Engine Guardrails Added After These Tables
### 2026-06-16 - Preserve Explicit Table Spans During Grid Cleanup
The `figure_caption_refine_composite` branch normalizes noisy flat table grids
by removing separator columns, duplicate columns, and sparse continuation rows.
That cleanup is useful for simple OCR table noise, but it is unsafe once a table
module has already emitted real `rowspan` or `colspan` structure. Rebuilding
from a plain grid would erase the very merged-header structure that the paper
metrics now score.
Implementation change:
- `normalize_table_grid_structure` now skips tables with explicit row/column
spans and records `normalized_table_grid_skipped_reason="explicit_spans"`.
- The table helper and candidate composite tests now verify that `colspan`
survives when grid normalization is enabled.
This is an engine guardrail, not a new accuracy result. It protects future
complex-table benchmarks from a known postprocessing failure mode.
### HF Recognizer Next V1 Admission
Generated artifact:
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-hf-recognizer-next-v1-combined-full/summary/summary.md)
- [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/k2-hf-recognizer-next-v1-combined-full/summary/summary.json)
- [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-hf-recognizer-next-v1-admission/validation.md)
- [validation config](/Users/cdjk/github/llm/ocr-tech/configs/validation-hf-recognizer-next-v1-admission.json)
Coverage:
- `244` held-out eval samples
- candidate, Tesseract, and stock PaddleOCR completed `244/244`
- Surya completed `243/244`; one English synthetic line crop timed out at `240s`
- admission validation status: `validated`
Headline paired CER table:
| baseline | slice | pairs | candidate CER | baseline CER | abs improvement | wins | losses | ties |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Tesseract | all | 180 | 0.226628 | 0.329012 | 0.102384 | 74 | 54 | 52 |
| Tesseract | Nepali | 30 | 0.633814 | 0.757175 | 0.123360 | 18 | 12 | 0 |
| stock PaddleOCR | all | 180 | 0.226628 | 0.656675 | 0.430048 | 159 | 10 | 11 |
| stock PaddleOCR | Nepali | 30 | 0.633814 | 0.945114 | 0.311300 | 24 | 2 | 4 |
| Surya | all | 179 | 0.223364 | 3.307235 | 3.083871 | 119 | 42 | 18 |
| Surya | Nepali | 30 | 0.633814 | 0.930927 | 0.297113 | 10 | 20 | 0 |
Interpretation:
- This is the first trained recognizer branch in the repo that clears a held-out OCR admission gate against Tesseract and stock PaddleOCR.
- Treat the Surya comparison as adapter-specific evidence. The English line-crop Surya CER is suspiciously high, and the Nepali slice has lower mean CER for the candidate but worse paired win count against Surya.
- This result supports "promising recognizer component"; it does not support "complete document-understanding engine" or SOTA.
### HF Recognizer Next V1 Table-Cell Refinement Smoke
Generated artifact:
- [model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/hf-table-cell-refine-next-v1/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-table-cell-refine-next-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-cell-refine-next-v1-smoke/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-table-cell-refine-next-v1-smoke/report.json)
One-sample hard-table smoke:
| system | CER | table_cell_f1 | table_teds_text | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| HF table-cell refine candidate | 0.363636 | 0.000000 | 0.294118 | 0.100000 | 9.888044 |
| Surya | 0.363636 | 1.000000 | 1.000000 | 0.100000 | 5.529877 |
Interpretation:
- The trained HF recognizer now runs as a leaf OCR engine inside the document pipeline.
- Unconditional HF replacement is not viable: on `eval-table-ledger-000-clean`, it replaced all `12` table cells and destroyed an already-correct Surya table.
- Do not scale this branch as-is. Any future HF table refinement needs a guarded replacement policy or a crop-specific recognizer, not forced replacement.
### Rendered Claim Pack
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-table/paper-table.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 25 | 0.248654 | 0.955556 | 0.764000 | 3.268937 |
| Surya | 25 | 0.195260 | 0.955556 | 0.750909 | 2.618872 |
| Tesseract | 25 | 0.689214 | 0.000000 | 0.816000 | 0.240963 |
| Stock Paddle | 25 | 1.749081 | 0.000000 | 1.000000 | 7.732034 |
Interpretation:
- Candidate beats Tesseract and stock-Paddle badly on text CER and table extraction.
- Candidate ties Surya on aggregate `table_cell_f1`.
- Candidate is slightly better than Surya on aggregate reading-order accuracy.
- Candidate still loses to raw Surya on text CER.
- Candidate is also slower than Tesseract and somewhat slower than Surya on this pack.
Latency caveat:
- these latency means come from separate benchmark reports on the same host and pack, then get pinned per system during table export
- treat them as comparable operational signals, not as a single synchronized runtime shootout
### Rendered Claim Pack V2 Holdout
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-table/paper-table.json)
- [preflight.md](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-script-router/preflight.md)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.180893 | 0.947500 | 0.750909 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 |
Interpretation:
- This is the stronger rendered-document holdout because calibration and holdout were separated at `50/50`.
- The routed candidate does not beat Surya here. It ties Surya exactly on text CER and reading-order accuracy.
- The routed candidate is slightly worse than raw Surya on aggregate `table_cell_f1`.
- The calibration artifact selected `primary_script_threshold=0.0`, which means the routed candidate collapsed to "always pick Surya" on the calibration split.
- That is why claim preflight now rejects this candidate as a routed model fit for a broad "better engine" claim.
### Rendered Claim Pack V2 Holdout - Surya Primary Latin Router
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-latin-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-latin-table/paper-table.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.416854 | 0.337500 | 0.761818 | 6.067455 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 6.699851 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.238464 |
Interpretation:
- This candidate was a Surya-primary, Latin-triggered page router with Tesseract fallback.
- It was meant to preserve Surya's text strength while using Tesseract for English-heavy pages.
- It failed badly on the rendered-v2 holdout: worse than raw Surya on text CER and much worse on table structure.
- Paired CER against Surya was decisive enough to rule the branch out: `0` wins, `12` losses, `38` ties.
### Rendered Claim Pack V2 Holdout - Surya Quality Router
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-quality-router-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-quality-router-table/paper-table.json)
- [calibration.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-quality-router/calibration.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.181128 | 0.947500 | 0.750909 | 4.620400 |
| Surya | 50 | 0.180893 | 0.947500 | 0.750909 | 5.603801 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.240773 |
Interpretation:
- This candidate was a Surya-vs-Tesseract quality selector calibrated on the rendered-v2 calibration split.
- Calibration selected `selected_threshold=0.0`, which means the quality gate kept Surya whenever it was not scored worse than Tesseract.
- On the untouched holdout, the candidate still did not beat raw Surya on text CER.
- Paired CER against Surya was effectively a tie with one bad miss: `0` wins, `1` loss, `49` ties.
- This is operationally closer to "Surya with a guardrail" than to a meaningfully new OCR engine.
### Rendered Claim Pack V2 Holdout - Line Align Composite
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-line-align-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-line-align-table/paper-table.json)
- [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-line-align-summary/summary.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.310549 | 0.897500 | 0.792727 | 4.142242 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 3.223913 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.235074 |
Interpretation:
- This candidate kept Surya as the text engine and reused Tesseract line scaffolding through `line_align_composite`.
- It did improve reading-order accuracy relative to raw Surya, and even edged past raw Tesseract on this holdout.
- That gain is not enough. CER regressed badly versus Surya, and aggregate `table_cell_f1` also regressed.
- Paired CER against Surya was one-sided enough to kill the branch: `0` wins, `7` losses, `43` ties.
- This is a useful negative result for the paper because it shows that better line ordering alone does not rescue the Surya/Tesseract composite family.
### Rendered Claim Pack V2 Holdout - Sample-Level Oracle
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-oracle-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-oracle-table/paper-table.json)
- [oracle-analysis.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-analysis.json)
- [oracle-choices.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-choices.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Oracle Surya/Tesseract | 50 | 0.180529 | 0.970833 | 0.750909 | 3.193115 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 3.223913 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.235074 |
Interpretation:
- This is not a deployable system. It is an upper-bound analysis that picks the better of raw Surya and raw Tesseract per sample using ground-truth CER.
- The oracle picked Surya for `49` of `50` holdout samples and Tesseract for only `1`.
- Per-sample oracle selections are recorded separately in `oracle-choices.json`; they are provenance, not benchmark metrics.
- The absolute CER gain over raw Surya is only `0.000364`.
- That means the current Surya/Tesseract routing space has almost no text-OCR headroom left on rendered-v2.
- Future work should move to a stronger recognizer/document model, not more Surya/Tesseract routing heuristics.
### Rendered Claim Pack V2 Holdout - Surya Error Pockets
Generated artifact:
- [error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-error-analysis/error-analysis.md)
- [top-failures.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json)
- [surya-vs-tesseract error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis/error-analysis.md)
- [surya-vs-oracle error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis/error-analysis.md)
- [eval-surya-hard15.jsonl](/Users/cdjk/github/llm/ocr-tech/data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl)
- [validation-hard15-surya.json](/Users/cdjk/github/llm/ocr-tech/configs/validation-hard15-surya.json)
- [line-align hard15 validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-claimpack-v2-line-align-hard15/validation.md)
- [k2 manifest-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-claimpack-v2-surya-hard15-k2/manifest-audit.md)
- [table/figure dev pack manifest](/Users/cdjk/github/llm/ocr-tech/data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl)
- [table/figure dev pack audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-eval-pack-table-figure-dev-v1-k2/eval-pack-audit.md)
Diagnostic summary:
| slice | sample_count | Surya CER |
| --- | ---: | ---: |
| figure | 20 | 0.253157 |
| table | 20 | 0.252781 |
| multi_page | 10 | 0.196472 |
| scan | 10 | 0.194861 |
| low_contrast | 10 | 0.193317 |
| blur | 10 | 0.190940 |
| clean | 10 | 0.182256 |
Interpretation:
- Surya's hardest absolute CER pockets on rendered-v2 are table-ledger and figure-caption samples, not generic blur/skew alone.
- Tesseract is worse than Surya on `49` of `50` samples and better on only `eval-form-notice-003-blur`; that single oracle win is too small to justify more Surya/Tesseract routing work.
- A ranked `15`-sample `hard_eval` manifest now exists for the next model branch; it contains `8` table samples and `7` figure samples, and its full file/reference audit passes on `k2`.
- `configs/validation-hard15-surya.json` defines the pocket gate a future candidate must clear against Surya, including intersection gates for `hard_eval,table` and `hard_eval,figure`.
- The current line-align candidate fails the hard15 gate: hard-eval CER is `0.482039` versus Surya `0.348706`, and hard table CER is `0.613636` versus Surya `0.363636`.
- A separate `80`-sample image development pack now exists for table/figure work: `40` table samples, `40` figure samples, all five degradations. It has zero sample-id overlap and zero text-hash overlap with hard15, and its `k2` audit passes.
- That development pack is not claim evidence; it is for model iteration before returning to the held-out hard15 and rendered-v2 gates.
- The next credible recognizer branch should target table/figure-heavy mixed-script pages and preserve Surya's already-strong behavior on the other `49` samples.
### Table/Figure Development Pack - HF TrOCR Negative Result
Generated artifact:
- [split-summary.md](/Users/cdjk/github/llm/ocr-tech/data/splits-table-figure-dev-v1/split-summary.md)
- [training_metrics.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-table-figure-dev-v1/training_metrics.json)
- [model-card.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-table-figure-dev-v1/package/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-table-figure-dev-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-figure-dev-v1/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-table-figure-dev-v1/benchmark/report.json)
- [paddle_recognizer.yml](/Users/cdjk/github/llm/ocr-tech/runs/paddle-table-figure-dev-v1/paddle_recognizer.yml)
Pack and split:
- `80` image samples: `40` table and `40` figure
- degradations: `clean`, `scan`, `blur`, `low_contrast`, `skew`
- split: `60` train / `20` eval
- grouping: `text_sha256`
- text groups: `16`
- split leakage: pass
- train/eval manifest audits on `k2`: pass
- hard15 overlap: `0` sample IDs and `0` text hashes
Training result for `microsoft/trocr-small-printed` after one epoch:
| metric | value |
| --- | ---: |
| eval_loss | 7.884517 |
| eval_cer | 3.062119 |
| eval_wer | 3.562689 |
| eval_exact_line_accuracy | 0.000000 |
Benchmark result on the `20`-sample dev eval:
| system | CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| HF table/figure candidate | 2.422001 | 0.000000 | 0.000000 |
| Surya | 0.563399 | 0.858333 | 0.550000 |
| Tesseract | 0.546227 | 0.000000 | 0.695000 |
Interpretation:
- The model package audits cleanly, so this is a quality failure rather than a packaging failure.
- Paired CER was `0` wins, `20` losses, and `0` ties against Surya.
- Paired CER was also `0` wins, `20` losses, and `0` ties against Tesseract.
- Naive full-page TrOCR fine-tuning is not a serious next step for table/figure-heavy document pages.
- The next model branch should use line/cell crops or a structure-aware page/document model rather than treating full document images as single recognition targets.
### Table/Figure Crop Development Pack - HF TrOCR Negative Result
Generated artifact:
- [crop-manifest-summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/crops-table-figure-dev-v1-k2/crop-manifest-summary.md)
- [crop manifest audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/crops-table-figure-dev-v1-k2/manifest-audit.md)
- [crop split-summary.md](/Users/cdjk/github/llm/ocr-tech/data/splits-crops-table-figure-dev-v1/split-summary.md)
- [training_metrics.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-crops-table-figure-dev-v1/training_metrics.json)
- [model-card.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-crops-table-figure-dev-v1/package/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-crops-table-figure-dev-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-crops-table-figure-dev-v1/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-crops-table-figure-dev-v1/benchmark/report.json)
Pack and split:
- source rendered samples: `80`
- extracted crops: `544`
- crop types: `384` table cells, `128` lines, `32` figure captions
- skipped candidates: `32` skew pages because pre-rotation bboxes are invalid
- split: `480` train / `64` eval
- grouping: `text_sha256`
- text groups: `68`
- crop manifest audit on `k2`: pass
- split leakage: pass
Training result for `microsoft/trocr-small-printed` after one crop-level epoch:
| metric | value |
| --- | ---: |
| eval_loss | 3.087561 |
| eval_cer | 0.959831 |
| eval_wer | 1.363715 |
| eval_exact_line_accuracy | 0.078125 |
Benchmark result on the `64`-sample crop eval:
| system | CER |
| --- | ---: |
| HF crop candidate | 1.043369 |
| Surya | 0.475309 |
| Tesseract | 0.077453 |
Interpretation:
- Crop-level training is a better problem shape than full-page recognition, but this recognizer is still not competitive.
- Paired CER against Surya was `7` wins, `50` losses, and `7` ties.
- Paired CER against Tesseract was `0` wins, `64` losses, and `0` ties.
- Tesseract is extremely strong on these rendered crop images, so near-term gains are more likely from better crop/layout detection plus strong crop OCR than from this small synthetic TrOCR fine-tune.
### Hard15 - Table Cell Refinement Composite
Generated artifact:
- [unsafe model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-v1/model-card.json)
- [unsafe summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-hard15-v1/summary.md)
- [unsafe cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-always-hard15-v1/table-cell-analysis.md)
- [safe model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-empty-v1/model-card.json)
- [safe summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-empty-hard15-v1/summary.md)
- [safe cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-empty-hard15-v1/table-cell-analysis.md)
- [safe validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-hard15-v1/validation.md)
- [safe+dedup model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-empty-dedup-v1/model-card.json)
- [safe+dedup summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-empty-dedup-hard15-v3/summary.md)
- [safe+dedup cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-empty-dedup-hard15-v3/table-cell-analysis.md)
- [safe+dedup validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-dedup-hard15-v3/validation.md)
Headline hard15 comparison:
| system | hard_eval CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| Surya baseline | 0.348706 | 0.979167 | 0.520000 |
| Table-cell refine, `always` | 0.348706 | 0.593750 | 0.520000 |
| Table-cell refine, `empty` | 0.348706 | 0.979167 | 0.520000 |
| Table-cell refine, `empty` + parser de-dup | 0.348706 | 0.979167 | 0.520000 |
Interpretation:
- The unsafe table-cell crop replacement policy damaged Surya's already-good table cells.
- The conservative empty-cell-only policy avoided regression but did not improve any hard15 metric.
- The safe policy's hard15 validation status is `not_validated`.
- The parser de-dup branch removes duplicate overlapping table exports for both candidate and Surya, so it improves artifact cleanliness but does not create a candidate advantage.
- Exact crop OCR being strong is not enough; the deployment problem is reliable cell detection and knowing when a replacement is safer than the structure engine's existing cell text.
Cell-level result:
- Surya is already exact on `94/96` hard15 reference table cells.
- Unsafe replacement is exact on only `57/96` cells.
- Unsafe replacement improves `0` cells over Surya and worsens `37`.
- Safe+dedup is exact on `94/96` cells, with `8` predicted tables for `8` reference tables and `0` duplicate overlapping tables on the v3 rerun.
### Hard15 - Figure Caption Refinement Composite
Generated artifact:
- [model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/figure-caption-refine-v1/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-figure-caption-refine-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-figure-caption-refine-hard15-v2/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-figure-caption-refine-hard15-v2/benchmark/report.json)
- [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-figure-caption-refine-hard15-v2/validation.md)
- [labels model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/figure-caption-refine-labels-v1/model-card.json)
- [labels summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-figure-caption-refine-labels-hard15-v1/summary.md)
- [labels validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-figure-caption-refine-labels-hard15-v1/validation.md)
Headline hard15 comparison after the parser caption-attachment fix:
| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
| Figure-caption refine | 0.228008 | 0.052478 | 1.000000 | 0.979167 | 0.520000 |
| Figure-caption refine + labels | 0.204610 | 0.008746 | 1.000000 | 0.979167 | 0.520000 |
Validation result:
- basic branch overall status: `not_validated`
- basic branch figure-caption gate: pass, `6` wins / `0` losses / `1` tie against Surya, improvement `0.039845`
- label branch overall status: `not_validated`
- label branch overall hard-eval CER gate: pass, improvement `0.032228`
- label branch figure CER gate: pass, improvement `0.069061`
- label branch figure-caption gate: pass, `7` wins / `0` losses / `0` ties against Surya, improvement `0.083576`
- label branch table gates: fail, ties against Surya
Interpretation:
- This is a real slice win over Surya on figure-caption extraction.
- It is not broad enough for a SOTA claim.
- Tesseract has better caption text but worse figure detection/table structure; the composite keeps Surya's structure and selectively borrows caption text.
- Safe empty-only replacement ties all `96` cells with Surya because there are no missing Surya cells in this pocket.
- The only remaining Surya cell errors are `किरण` predicted as `करिण` on `eval-table-ledger-002-clean` and `eval-table-ledger-002-scan`.
### Rendered Blur-Only Expensive-Baseline Subset
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-blur5-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-blur5-table/paper-table.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 5 | 0.223273 | 1.000000 | 0.783636 | 3.784279 |
| Surya | 5 | 0.197601 | 1.000000 | 0.750909 | 2.943875 |
| Tesseract | 5 | 0.732364 | 0.000000 | 0.823636 | 0.220515 |
| Stock Paddle | 5 | 2.810426 | 0.000000 | 1.000000 | 7.440482 |
| GLM-OCR | 5 | 2.820269 | 0.000000 | 1.000000 | 16.846619 |
| PaddleOCR-VL | 5 | 18.470663 | 0.000000 | 0.750909 | 32.793166 |
Interpretation:
- Candidate is materially stronger than Tesseract, stock-Paddle, GLM-OCR, and PaddleOCR-VL on this degraded rendered subset.
- Candidate still does not beat raw Surya on text CER.
- The external VLM-style baselines are also much slower than the current candidate here.
## Branch Chronology
This is the shortest honest story of the project so far.
| branch | eval setting | candidate CER | baseline CER | paired result | verdict | evidence |
| --- | --- | ---: | ---: | --- | --- | --- |
| HF TrOCR recognizer | mixed Nepali/English line-style holdout, 180 samples | 0.586139 | 0.329012 vs Tesseract | 36 wins, 97 losses, 47 ties | runnable but not competitive | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-hf-vs-tesseract-summary/summary.json) |
| Tesseract + ByT5 corrector | same 180-sample mixed holdout | 6.912065 | 0.329012 vs Tesseract | 0 wins, 170 losses, 10 ties | failed badly | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-corrector-vs-tesseract-summary/summary.md) |
| quality router (`psm=6` vs `psm=13`) | 91-sample mixed holdout | 0.340608 | 0.342234 vs Tesseract | 2 wins, 1 loss, 88 ties | tiny gain, plausible baseline only | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-quality-router-holdout/summary.json) |
| ranked PSM ensemble (`6/7/11/13`) | same 91-sample mixed holdout | 0.358352 | 0.342234 vs Tesseract | 9 wins, 16 losses, 66 ties | overfit, worse than baseline | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-psm-ensemble-holdout/summary.json) |
| script router (`tesseract -> surya`) | same 91-sample mixed holdout | 0.313084 | 0.342234 vs Tesseract | 9 wins, 0 losses, 82 ties | strongest older line-style result | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-script-router-holdout/summary.json) |
| script router + structure backfill | rendered claim-ready document pack, 25 samples | 0.248654 | 0.689214 vs Tesseract | 9 wins, 1 loss, 15 ties | strong document-pack improvement over Tesseract | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-rendered-backfill/summary.json) |
| raw Surya on rendered claim pack | rendered claim-ready document pack, 25 samples | 0.195260 | 0.248654 vs Candidate | candidate loses on text CER | current text leader on rendered pack | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-surya-compare/summary.json) |
| rendered-v2 script router | rendered claim-ready document holdout, 50 samples | 0.180893 | 0.180893 vs Surya | exact CER tie, worse table structure | routed candidate collapses to Surya on calibration; not a meaningful new engine | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-script-router/summary.json) |
| rendered-v2 Surya-primary Latin router | rendered claim-ready document holdout, 50 samples | 0.416854 | 0.180893 vs Surya | 0 wins, 12 losses, 38 ties on CER | failed badly; inverted script routing is not a viable answer | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-latin-router/summary.json) |
| rendered-v2 Surya quality router | rendered claim-ready document holdout, 50 samples | 0.181128 | 0.180893 vs Surya | 0 wins, 1 loss, 49 ties on CER | nearly indistinguishable from Surya and still not better | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-quality-router/summary.json) |
| rendered-v2 line-align composite | rendered claim-ready document holdout, 50 samples | 0.310549 | 0.180893 vs Surya | 0 wins, 7 losses, 43 ties on CER | better reading order, but text and table regressions are too large | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-line-align-summary/summary.json) |
| rendered-v2 Surya/Tesseract sample oracle | rendered claim-ready document holdout, 50 samples | 0.180529 | 0.180893 vs Surya | oracle picks Tesseract only 1/50 samples | upper bound shows routing headroom is essentially exhausted | [oracle-analysis.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-analysis.json) |
| HF TrOCR table/figure dev | table/figure development eval, 20 samples | 2.422001 | 0.563399 vs Surya; 0.546227 vs Tesseract | 0 wins, 20 losses, 0 ties vs both | failed badly; full-page TrOCR fine-tuning is not viable here | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-figure-dev-v1/summary.md) |
| HF TrOCR crop dev | table/figure crop eval, 64 samples | 1.043369 | 0.475309 vs Surya; 0.077453 vs Tesseract | 7 wins, 50 losses, 7 ties vs Surya; 0 wins, 64 losses vs Tesseract | crop shape helps but this recognizer still fails | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-crops-table-figure-dev-v1/summary.md) |
| table-cell refine, unsafe | hard15 rendered-v2 pocket, 15 samples | 0.348706 | 0.348706 vs Surya | text tie, table F1 regresses from 0.979167 to 0.593750 | replacement policy is unsafe | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-hard15-v1/summary.md) |
| table-cell refine, empty-only | hard15 rendered-v2 pocket, 15 samples | 0.348706 | 0.348706 vs Surya | exact tie across CER/table/reading order; validation not_validated | safe but no improvement | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-hard15-v1/validation.md) |
| normalized script router, hard-timeout rerun | normalized mixed Nepali/English line-style eval, 180 samples | 0.354840 | 0.329012 vs Tesseract; 3.157317 vs Surya on 178 scored Surya rows | 17 wins, 4 losses, 159 ties on CER vs Tesseract but worse aggregate mean; 2 Surya rows timed out at 45s | not a better engine; benchmark watchdog works and routing still loses overall CER | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-mixed-v1-normalized-script-router-hard-timeout/outputs/summary-mixed-v1-normalized-script-router-hard-timeout/summary.md) |
| PaddleOCR recognizer hard-50 failure slice | top 50 failures from AWS G6 trained PaddleOCR recognizer, remapped to normalized k2 eval paths | n/a | n/a | 50 failures: 38 English line-crop synthetic, 12 Nepali synthetic; k2 manifest audit passed | next recognizer gate; do not train on it | [manifest-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-paddle-mixed-v2-hard50-normalized/outputs/manifest-audit-paddle-mixed-v2-cuda-hard50-normalized/manifest-audit.md) |
| PaddleOCR hard-50 gate negative control | failed AWS G6 trained PaddleOCR recognizer evaluated on the hard-50 gate | 0.887557 | 0.080249 vs Tesseract | fails all four hard gates; English line-crop CER `0.852048` vs `0.031244` | validates the gate catches the known recognizer collapse | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-paddle-mixed-v2-cuda-hard50-tesseract-negative/validation.md) |
| line-level Tesseract/Paddle composite | hard-50 failure slice, Tesseract primary with PaddleOCR secondary only for Devanagari-looking lines | 0.276489 | 0.120106 vs Tesseract | English line-crop ties exactly (`0.060429` CER); Nepali collapses (`0.960679` vs `0.309083` CER, `1.000000` vs `0.711607` WER) | safe English preservation but failed Devanagari override; do not promote this recognizer | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-line-script-paddle-hard50-v2/validation.md) |
| PaddleOCR v3 CUDA recognizer | G6-trained lower-LR PaddleOCR branch on hard-50 admission gate | 0.921416 | 0.080249 vs Tesseract | 0 wins, 50 losses on CER; English line-crop CER `0.917511` vs `0.031244`; Nepali WER `1.055556` vs `0.716369` | failed worse than v2; stop branch before full mixed eval | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/aws-g6-paddle-mixed-v3-cuda/outputs/validation-paddle-mixed-v3-cuda-hard50-rescored/validation.md) |
| PaddleOCR v4 CUDA recognizer | G6-trained Devanagari PP-OCRv3 branch, LR `0.00005`, 20 epochs, evaluated on hard-50 admission gate | 0.940500 | 0.080249 vs Tesseract | 0 wins, 50 losses on CER; English line-crop CER `0.950730` vs `0.031244`; Nepali CER `0.908108` vs `0.235433` | failed worse than v2/v3; the same PaddleOCR mixed-script fine-tune branch should not be the engine core | [artifact bundle](/Users/cdjk/github/llm/ocr-tech/outputs/aws-g6-paddle-mixed-v4-cuda/ocrtech-paddle-mixed-v4-cuda-artifacts.tgz) |
## Current Best-Evidence Statement
The strongest evidence-backed statement available today is:
1. The current Tesseract-plus-Surya backfill candidate is a real document-OCR improvement over Tesseract and several weaker baselines on internal rendered Nepali/English document packs.
2. On the larger rendered-v2 holdout, every Tesseract/Surya composite tested so far either collapses to Surya, trails Surya, becomes effectively indistinguishable from Surya, or trades too much OCR accuracy for modest ordering gains.
3. On the cleaned 180-sample mixed line-style eval with hard per-sample timeouts, the older script-router candidate also loses to Tesseract overall (`0.354840` vs `0.329012` CER), despite a small overall WER improvement (`0.469944` vs `0.486485`) and a Nepali WER improvement.
4. The trained PaddleOCR recognizer failure profile is now materialized as a normalized hard-50 eval gate: all top failures are synthetic line crops, mostly English (`38/50`), so the next recognizer branch must prove English preservation instead of only improving Nepali.
5. The line-level Tesseract/Paddle composite proves the candidate shape can preserve English while routing Devanagari-looking lines, but the available trained PaddleOCR recognizer is too weak on the Nepali hard slice to use as an override (`0.960679` vs Tesseract `0.309083` CER).
6. The second AWS G6 PaddleOCR attempt fixed recipe/runtime issues but failed the hard-50 gate even worse than the first (`0.921416` vs Tesseract `0.080249` CER), so this PP-OCRv3 Devanagari recipe family is not the engine path.
7. A sample-level oracle over raw Surya and raw Tesseract has almost no text-OCR headroom over Surya (`0.180529` vs `0.180893` CER), so more routing in this component family is not a serious path to SOTA.
8. The repo now has enough validation discipline and artifact traceability to support a serious comparative paper, but not a SOTA claim.
## Regeneration Workflow
Rebuild the generated comparison tables with:
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-claimpack-rendered-backfill/report.json \
--out outputs/paper-claimpack-summary-tesseract \
--candidate candidate \
--baseline tesseract \
--metric cer
```
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-claimpack-surya-compare/report.json \
--out outputs/paper-claimpack-summary-surya \
--candidate candidate \
--baseline surya \
--metric cer
```
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-claimpack-stock-paddle-fixed/report.json \
--out outputs/paper-claimpack-summary-stock-paddle \
--candidate candidate \
--baseline stock-paddle \
--metric cer
```
```bash
ocrtech paper-benchmark-table \
--summary outputs/paper-claimpack-summary-tesseract/summary.json \
--summary outputs/paper-claimpack-summary-surya/summary.json \
--summary outputs/paper-claimpack-summary-stock-paddle/summary.json \
--out outputs/paper-claimpack-table \
--system candidate \
--system surya \
--system tesseract \
--system stock-paddle \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--label 'stock-paddle=Stock Paddle' \
--source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' \
--source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' \
--source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' \
--source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds
```
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-rendered-blur5/report.json \
--out outputs/paper-blur5-summary \
--candidate candidate \
--baseline tesseract \
--baseline surya \
--baseline stock-paddle \
--baseline glm-ocr \
--baseline paddleocr-vl \
--metric cer
```
```bash
ocrtech paper-benchmark-table \
--summary outputs/paper-blur5-summary/summary.json \
--out outputs/paper-blur5-table \
--system candidate \
--system surya \
--system tesseract \
--system stock-paddle \
--system glm-ocr \
--system paddleocr-vl \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--label 'stock-paddle=Stock Paddle' \
--label 'glm-ocr=GLM-OCR' \
--label 'paddleocr-vl=PaddleOCR-VL' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds
```
```bash
ocrtech paper-benchmark-table \
--summary outputs/live-k2-claimpack-v2-script-router/summary.json \
--out outputs/paper-claimpack-v2-table \
--system candidate \
--system surya \
--system tesseract \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--title 'Rendered Claim Pack V2 Holdout Comparison'
```
```bash
ocrtech paper-benchmark-table \
--summary outputs/live-k2-claimpack-v2-surya-quality-router/summary.json \
--out outputs/paper-claimpack-v2-surya-quality-router-table \
--system candidate \
--system surya \
--system tesseract \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds \
--title 'Rendered Claim Pack V2 Holdout - Surya Quality Router'
```
```bash
ocrtech paper-benchmark-table \
--summary outputs/live-k2-claimpack-v2-line-align-summary/summary.json \
--out outputs/paper-claimpack-v2-line-align-table \
--system candidate \
--system surya \
--system tesseract \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds \
--title 'Rendered Claim Pack V2 Holdout - Line Align Composite'
```
```bash
ocrtech benchmark-oracle \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--out outputs/paper-claimpack-v2-surya-tesseract-oracle \
--system surya \
--system tesseract \
--metric cer \
--oracle-name oracle-surya-tesseract
ocrtech paper-benchmark-table \
--summary outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-summary/summary.json \
--out outputs/paper-claimpack-v2-oracle-table \
--system oracle-surya-tesseract \
--system surya \
--system tesseract \
--label 'oracle-surya-tesseract=Oracle Surya/Tesseract' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds \
--title 'Rendered Claim Pack V2 Holdout - Sample-Level Oracle'
```
```bash
ocrtech benchmark-error-analysis \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--out outputs/paper-claimpack-v2-surya-error-analysis \
--system surya \
--metric cer \
--top-n 15
ocrtech benchmark-error-analysis \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--out outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis \
--system surya \
--baseline tesseract \
--metric cer \
--top-n 15
ocrtech benchmark-error-analysis \
--benchmark-report outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-report.json \
--out outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis \
--system surya \
--baseline oracle-surya-tesseract \
--metric cer \
--top-n 10
ocrtech export-failure-manifest \
--source-manifest data/splits-claimpack-rendered-v2/eval.jsonl \
--top-failures outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json \
--out data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
--split hard_eval
ocrtech validate-claim \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--config configs/validation-hard15-surya.json \
--eval-manifest data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
--out outputs/validation-claimpack-v2-line-align-hard15
ocrtech create-eval-pack \
--out data/eval-pack-table-figure-dev-v1 \
--input-format image \
--template table-ledger \
--template figure-caption \
--count-per-template 8 \
--variant-offset 100 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast \
--degradation skew
```
Memory still is not part of these generated tables. The current benchmark artifact format does not record it.
## Hard15 Figure-Caption + Table-Lexicon Candidate
Latest artifact set:
- `outputs/figure-caption-table-lexicon-v1/model-card.json`
- `outputs/experiment-figure-caption-table-lexicon-hard15-v3/benchmark/report.json`
- `outputs/summary-figure-caption-table-lexicon-hard15-v3/summary.md`
- `outputs/table-cell-analysis-figure-caption-table-lexicon-hard15-v3/table-cell-analysis.md`
- `outputs/validation-figure-caption-table-lexicon-hard15-v3/validation.md`
Headline table for internal hard15 only:
| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| candidate | 0.010671 | 0.008746 | 1.000000 | 1.000000 | 0.520000 |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
Validation status is still `not_validated`.
Passed configured Surya gates:
- overall hard_eval CER: `15` wins / `0` losses / `0` ties
- table-slice CER: `8` wins / `0` losses / `0` ties
- figure-slice CER: `7` wins / `0` losses / `0` ties
- figure-caption CER: `7` wins / `0` losses / `0` ties
Remaining configured failure:
- table-cell F1: candidate `1.000000`, Surya `0.979167`, but bootstrap CI low is `0.000000` because there are only 2 improving table pages and 6 ties
Cell-level table evidence:
| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 96 | 96 | 1.000000 | 0.000000 |
| Surya | 96 | 94 | 0.979167 | 0.010417 |
| Tesseract | 96 | 0 | 0.000000 | 1.000000 |
Use this result as an internal milestone, not a paper claim of broad SOTA. The candidate is lexicon-assisted for table names and uses guarded structural-title corrections (`वार्षिक प्रतिवेदन`, `दैनिक हिसाब`, `चित्र`, `निष्कर्ष`). A publishable table claim needs a larger independent table pack so confidence intervals are meaningful.
## Targeted Table-Ledger Validation Pack V2
Artifacts:
- `data/eval-pack-table-ledger-claim-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-table-ledger-claim-v1/eval-pack-audit.md`
- `outputs/experiment-table-ledger-claim-v2/benchmark/report.json`
- `outputs/summary-table-ledger-claim-v2/summary.md`
- `outputs/table-cell-analysis-table-ledger-claim-v2/table-cell-analysis.md`
- `outputs/validation-table-ledger-claim-v2b/validation.md`
- `configs/validation-table-ledger-surya.json`
Pack:
- template: `table-ledger`
- samples: `60`
- cells: `720`
- degradations: clean, scan, blur, low_contrast, skew
- variant offset: `300`
- claim-ready broad coverage: no, targeted table-only pack
Validation status: `validated`.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 60 | 15 | 0 | 45 | 1.000000 | 0.979167 | 0.020833 | 0.012500 |
| table_exact_match | 60 | 15 | 0 | 45 | 1.000000 | 0.750000 | 0.250000 | 0.150000 |
| table_structure_similarity | 60 | 0 | 0 | 60 | 1.000000 | 1.000000 | 0.000000 | 0.000000 |
Cell exactness:
| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 720 | 720 | 1.000000 | 0.000000 |
| Surya | 720 | 705 | 0.979167 | 0.005208 |
| Tesseract | 720 | 0 | 0.000000 | 0.999868 |
Runtime on k2:
| system | mean latency seconds | median latency seconds |
| --- | ---: | ---: |
| candidate | 3.017929 | 2.967943 |
| Surya | 2.478481 | 2.453125 |
| Tesseract | 0.146866 | 0.134735 |
Paper-safe claim text:
> On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved structured table extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, and passed bootstrap-gated validation for both table-cell F1 and table exact match.
More precise paper-safe wording:
> On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved table cell-content extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, while matching Surya's perfect grid-structure score.
Do not generalize this to all tables. The pack has no merged cells, no nested headers, and no real scanned government tables. The scoring path now supports span-aware structured references, but this pack does not exercise that capability and the metric is still not true HTML-tree TEDS.
## Targeted Complex-Table Validation Pack V1
Artifacts:
- `data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-complex-table-v1/eval-pack-audit.md`
- `outputs/experiment-complex-table-v1/benchmark/report.json`
- `outputs/summary-complex-table-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-v1/table-cell-analysis.md`
- `outputs/validation-complex-table-v1/validation.md`
- `configs/validation-complex-table-surya.json`
Pack:
- template: `complex-table-ledger`
- samples: `40`
- reference cells: `600`
- span reference: merged header with `colspan=2`
- degradations: clean, scan, blur, low_contrast
- broad claim-ready coverage: no
Validation status: `validated` for cell-content improvement plus structure non-regression.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 8 | 0 | 32 | 0.928333 | 0.915000 | 0.013333 | 0.005000 |
| table_exact_match | 40 | 0 | 0 | 40 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| table_structure_similarity | 40 | 0 | 0 | 40 | 0.875000 | 0.875000 | 0.000000 | 0.000000 |
Cell exactness:
| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 557 | 0.928333 | 0.067667 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |
Paper-safe claim text:
> On a targeted generated complex-table pack with span-aware references, the lexicon-assisted candidate improved table cell-content F1 over Surya but did not improve merged-header reconstruction; both systems failed exact table match and tied on span-aware structure similarity.
This is a negative result for complex table structure. The next model/parser branch must preserve or infer rowspans/colspans before we can claim complex-table document understanding.
## Targeted Complex-Table Span-Inference Pack V1
After adding conservative repeated-header colspan inference, we reran the same `40`-sample generated complex-table pack on k2.
Artifacts:
- `outputs/experiment-complex-table-span-infer-v1/benchmark/report.json`
- `outputs/experiment-complex-table-span-infer-v1/validation/validation.md`
- `outputs/summary-complex-table-span-infer-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-span-infer-v1/table-cell-analysis.md`
Validation status: `validated`.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |
Cell exactness:
| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 597 | 0.995000 | 0.001000 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |
Paper-safe claim text:
> On a targeted generated complex-table pack with one repeated merged-header pattern, a conservative span-inference postprocessor improved table structure extraction over Surya, raising span-aware structure similarity from 0.875 to 1.000 and table exact match from 0.000 to 0.925.
Do not generalize this to all complex tables. This is generated data with a single span pattern and a heuristic parser repair. The next paper-grade check needs real scanned tables with varied merged cells, rowspans, nested headers, and a true HTML/TEDS scorer.
## Targeted Complex-Table TEDS-Style Metrics V1
We added deterministic ordered table-tree edit metrics and reran the same complex-table pack with these gates included.
Artifacts:
- `outputs/experiment-complex-table-teds-v1/benchmark/report.json`
- `outputs/experiment-complex-table-teds-v1/validation/validation.md`
- `outputs/summary-complex-table-teds-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-teds-v1/table-cell-analysis.md`
Validation status: `validated`.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |
| table_teds_structure | 40 | 40 | 0 | 0 | 1.000000 | 0.904762 | 0.095238 | 0.095238 |
| table_teds_text | 40 | 40 | 0 | 0 | 0.996250 | 0.891667 | 0.104583 | 0.098690 |
Paper-safe claim text:
> On a targeted generated complex-table pack, the candidate improved ordered table-tree metrics over Surya, reaching 1.000 structure-only TEDS-style similarity and 0.996 text-sensitive TEDS-style similarity, compared with Surya's 0.905 and 0.892 respectively.
This is still not a public-benchmark SOTA claim. The metrics are deterministic and closer to table-structure scoring, but the pack is generated and covers one repeated-header merged-cell pattern.
## HF Recognizer Next V1 Admission Benchmark
The AWS G6 trained HF recognizer was evaluated on the `recognizer-next-v1`
held-out eval manifest against Tesseract on `k2`.
Artifacts:
- `outputs/k2-hf-recognizer-next-v1-full-cpu/benchmark/report.json`
- `outputs/k2-hf-recognizer-next-v1-full-cpu/summary/summary.md`
- candidate model card:
`outputs/aws-g6-hf-recognizer-next-v1/package/model-card-cpu.json`
Run status:
- candidate rows: `244` ok, `0` errors
- Tesseract rows: `244` ok, `0` errors
- paired CER rows with text references: `180`
| slice | pairs | candidate CER | Tesseract CER | improvement | wins | losses | ties |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| all | 180 | 0.226628 | 0.329012 | 0.102384 | 74 | 54 | 52 |
| english/en/line_crop | 150 | 0.145190 | 0.243379 | 0.098189 | 56 | 42 | 52 |
| nepali | 30 | 0.633814 | 0.757175 | 0.123360 | 18 | 12 | 0 |
| real | 15 | 0.832255 | 1.162917 | 0.330662 | 12 | 3 | 0 |
| synthetic | 165 | 0.171571 | 0.253202 | 0.081631 | 62 | 51 | 52 |
Paper-safe claim text:
> On the held-out `recognizer-next-v1` text-reference slices, the trained HF
> recognizer improves paired CER over Tesseract, reducing aggregate CER from
> 0.329 to 0.227 across 180 paired samples.
Do not call this SOTA. The result is a recognizer admission benchmark against
Tesseract only; the branch still needs Surya, stock PaddleOCR, GLM-OCR, public
document benchmarks, table structure metrics, reading-order validation, and
document reconstruction gates.