Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified |
Download ocr-tech-mps-integration/docs/PAPER_RESULTS.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 54.3 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/PAPER_RESULTS.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech-mps-integration/docs/PAPER_RESULTS.md
-
curl -L -o PAPER_RESULTS.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/PAPER_RESULTS.md
54.3 kB
| # Paper Results | |
| This file is the shortest path from repo artifacts to a paper draft. | |
| Use [PAPER_NOTES.md](PAPER_NOTES.md) for claim boundaries and framing. | |
| Use this file for the concrete result tables and the branch chronology. | |
| ## Current Comparison Tables | |
| ## Engine Guardrails Added After These Tables | |
| ### 2026-06-16 - Preserve Explicit Table Spans During Grid Cleanup | |
| The `figure_caption_refine_composite` branch normalizes noisy flat table grids | |
| by removing separator columns, duplicate columns, and sparse continuation rows. | |
| That cleanup is useful for simple OCR table noise, but it is unsafe once a table | |
| module has already emitted real `rowspan` or `colspan` structure. Rebuilding | |
| from a plain grid would erase the very merged-header structure that the paper | |
| metrics now score. | |
| Implementation change: | |
| - `normalize_table_grid_structure` now skips tables with explicit row/column | |
| spans and records `normalized_table_grid_skipped_reason="explicit_spans"`. | |
| - The table helper and candidate composite tests now verify that `colspan` | |
| survives when grid normalization is enabled. | |
| This is an engine guardrail, not a new accuracy result. It protects future | |
| complex-table benchmarks from a known postprocessing failure mode. | |
| ### HF Recognizer Next V1 Admission | |
| Generated artifact: | |
| - [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-hf-recognizer-next-v1-combined-full/summary/summary.md) | |
| - [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/k2-hf-recognizer-next-v1-combined-full/summary/summary.json) | |
| - [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-hf-recognizer-next-v1-admission/validation.md) | |
| - [validation config](/Users/cdjk/github/llm/ocr-tech/configs/validation-hf-recognizer-next-v1-admission.json) | |
| Coverage: | |
| - `244` held-out eval samples | |
| - candidate, Tesseract, and stock PaddleOCR completed `244/244` | |
| - Surya completed `243/244`; one English synthetic line crop timed out at `240s` | |
| - admission validation status: `validated` | |
| Headline paired CER table: | |
| | baseline | slice | pairs | candidate CER | baseline CER | abs improvement | wins | losses | ties | | |
| | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | |
| | Tesseract | all | 180 | 0.226628 | 0.329012 | 0.102384 | 74 | 54 | 52 | | |
| | Tesseract | Nepali | 30 | 0.633814 | 0.757175 | 0.123360 | 18 | 12 | 0 | | |
| | stock PaddleOCR | all | 180 | 0.226628 | 0.656675 | 0.430048 | 159 | 10 | 11 | | |
| | stock PaddleOCR | Nepali | 30 | 0.633814 | 0.945114 | 0.311300 | 24 | 2 | 4 | | |
| | Surya | all | 179 | 0.223364 | 3.307235 | 3.083871 | 119 | 42 | 18 | | |
| | Surya | Nepali | 30 | 0.633814 | 0.930927 | 0.297113 | 10 | 20 | 0 | | |
| Interpretation: | |
| - This is the first trained recognizer branch in the repo that clears a held-out OCR admission gate against Tesseract and stock PaddleOCR. | |
| - Treat the Surya comparison as adapter-specific evidence. The English line-crop Surya CER is suspiciously high, and the Nepali slice has lower mean CER for the candidate but worse paired win count against Surya. | |
| - This result supports "promising recognizer component"; it does not support "complete document-understanding engine" or SOTA. | |
| ### HF Recognizer Next V1 Table-Cell Refinement Smoke | |
| Generated artifact: | |
| - [model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/hf-table-cell-refine-next-v1/model-card.json) | |
| - [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-table-cell-refine-next-v1-model/model-audit.md) | |
| - [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-cell-refine-next-v1-smoke/summary.md) | |
| - [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-table-cell-refine-next-v1-smoke/report.json) | |
| One-sample hard-table smoke: | |
| | system | CER | table_cell_f1 | table_teds_text | reading_order_pair_accuracy | latency_seconds | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | HF table-cell refine candidate | 0.363636 | 0.000000 | 0.294118 | 0.100000 | 9.888044 | | |
| | Surya | 0.363636 | 1.000000 | 1.000000 | 0.100000 | 5.529877 | | |
| Interpretation: | |
| - The trained HF recognizer now runs as a leaf OCR engine inside the document pipeline. | |
| - Unconditional HF replacement is not viable: on `eval-table-ledger-000-clean`, it replaced all `12` table cells and destroyed an already-correct Surya table. | |
| - Do not scale this branch as-is. Any future HF table refinement needs a guarded replacement policy or a crop-specific recognizer, not forced replacement. | |
| ### Rendered Claim Pack | |
| Generated artifact: | |
| - [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-table/paper-table.md) | |
| - [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-table/paper-table.json) | |
| Headline table: | |
| | system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | Candidate | 25 | 0.248654 | 0.955556 | 0.764000 | 3.268937 | | |
| | Surya | 25 | 0.195260 | 0.955556 | 0.750909 | 2.618872 | | |
| | Tesseract | 25 | 0.689214 | 0.000000 | 0.816000 | 0.240963 | | |
| | Stock Paddle | 25 | 1.749081 | 0.000000 | 1.000000 | 7.732034 | | |
| Interpretation: | |
| - Candidate beats Tesseract and stock-Paddle badly on text CER and table extraction. | |
| - Candidate ties Surya on aggregate `table_cell_f1`. | |
| - Candidate is slightly better than Surya on aggregate reading-order accuracy. | |
| - Candidate still loses to raw Surya on text CER. | |
| - Candidate is also slower than Tesseract and somewhat slower than Surya on this pack. | |
| Latency caveat: | |
| - these latency means come from separate benchmark reports on the same host and pack, then get pinned per system during table export | |
| - treat them as comparable operational signals, not as a single synchronized runtime shootout | |
| ### Rendered Claim Pack V2 Holdout | |
| Generated artifact: | |
| - [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-table/paper-table.md) | |
| - [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-table/paper-table.json) | |
| - [preflight.md](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-script-router/preflight.md) | |
| Headline table: | |
| | system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | | |
| | --- | ---: | ---: | ---: | ---: | | |
| | Candidate | 50 | 0.180893 | 0.947500 | 0.750909 | | |
| | Surya | 50 | 0.180893 | 0.970833 | 0.750909 | | |
| | Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | | |
| Interpretation: | |
| - This is the stronger rendered-document holdout because calibration and holdout were separated at `50/50`. | |
| - The routed candidate does not beat Surya here. It ties Surya exactly on text CER and reading-order accuracy. | |
| - The routed candidate is slightly worse than raw Surya on aggregate `table_cell_f1`. | |
| - The calibration artifact selected `primary_script_threshold=0.0`, which means the routed candidate collapsed to "always pick Surya" on the calibration split. | |
| - That is why claim preflight now rejects this candidate as a routed model fit for a broad "better engine" claim. | |
| ### Rendered Claim Pack V2 Holdout - Surya Primary Latin Router | |
| Generated artifact: | |
| - [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-latin-table/paper-table.md) | |
| - [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-latin-table/paper-table.json) | |
| Headline table: | |
| | system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | Candidate | 50 | 0.416854 | 0.337500 | 0.761818 | 6.067455 | | |
| | Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 6.699851 | | |
| | Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.238464 | | |
| Interpretation: | |
| - This candidate was a Surya-primary, Latin-triggered page router with Tesseract fallback. | |
| - It was meant to preserve Surya's text strength while using Tesseract for English-heavy pages. | |
| - It failed badly on the rendered-v2 holdout: worse than raw Surya on text CER and much worse on table structure. | |
| - Paired CER against Surya was decisive enough to rule the branch out: `0` wins, `12` losses, `38` ties. | |
| ### Rendered Claim Pack V2 Holdout - Surya Quality Router | |
| Generated artifact: | |
| - [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-quality-router-table/paper-table.md) | |
| - [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-quality-router-table/paper-table.json) | |
| - [calibration.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-quality-router/calibration.json) | |
| Headline table: | |
| | system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | Candidate | 50 | 0.181128 | 0.947500 | 0.750909 | 4.620400 | | |
| | Surya | 50 | 0.180893 | 0.947500 | 0.750909 | 5.603801 | | |
| | Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.240773 | | |
| Interpretation: | |
| - This candidate was a Surya-vs-Tesseract quality selector calibrated on the rendered-v2 calibration split. | |
| - Calibration selected `selected_threshold=0.0`, which means the quality gate kept Surya whenever it was not scored worse than Tesseract. | |
| - On the untouched holdout, the candidate still did not beat raw Surya on text CER. | |
| - Paired CER against Surya was effectively a tie with one bad miss: `0` wins, `1` loss, `49` ties. | |
| - This is operationally closer to "Surya with a guardrail" than to a meaningfully new OCR engine. | |
| ### Rendered Claim Pack V2 Holdout - Line Align Composite | |
| Generated artifact: | |
| - [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-line-align-table/paper-table.md) | |
| - [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-line-align-table/paper-table.json) | |
| - [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-line-align-summary/summary.json) | |
| Headline table: | |
| | system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | Candidate | 50 | 0.310549 | 0.897500 | 0.792727 | 4.142242 | | |
| | Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 3.223913 | | |
| | Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.235074 | | |
| Interpretation: | |
| - This candidate kept Surya as the text engine and reused Tesseract line scaffolding through `line_align_composite`. | |
| - It did improve reading-order accuracy relative to raw Surya, and even edged past raw Tesseract on this holdout. | |
| - That gain is not enough. CER regressed badly versus Surya, and aggregate `table_cell_f1` also regressed. | |
| - Paired CER against Surya was one-sided enough to kill the branch: `0` wins, `7` losses, `43` ties. | |
| - This is a useful negative result for the paper because it shows that better line ordering alone does not rescue the Surya/Tesseract composite family. | |
| ### Rendered Claim Pack V2 Holdout - Sample-Level Oracle | |
| Generated artifact: | |
| - [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-oracle-table/paper-table.md) | |
| - [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-oracle-table/paper-table.json) | |
| - [oracle-analysis.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-analysis.json) | |
| - [oracle-choices.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-choices.json) | |
| Headline table: | |
| | system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | Oracle Surya/Tesseract | 50 | 0.180529 | 0.970833 | 0.750909 | 3.193115 | | |
| | Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 3.223913 | | |
| | Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.235074 | | |
| Interpretation: | |
| - This is not a deployable system. It is an upper-bound analysis that picks the better of raw Surya and raw Tesseract per sample using ground-truth CER. | |
| - The oracle picked Surya for `49` of `50` holdout samples and Tesseract for only `1`. | |
| - Per-sample oracle selections are recorded separately in `oracle-choices.json`; they are provenance, not benchmark metrics. | |
| - The absolute CER gain over raw Surya is only `0.000364`. | |
| - That means the current Surya/Tesseract routing space has almost no text-OCR headroom left on rendered-v2. | |
| - Future work should move to a stronger recognizer/document model, not more Surya/Tesseract routing heuristics. | |
| ### Rendered Claim Pack V2 Holdout - Surya Error Pockets | |
| Generated artifact: | |
| - [error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-error-analysis/error-analysis.md) | |
| - [top-failures.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json) | |
| - [surya-vs-tesseract error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis/error-analysis.md) | |
| - [surya-vs-oracle error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis/error-analysis.md) | |
| - [eval-surya-hard15.jsonl](/Users/cdjk/github/llm/ocr-tech/data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl) | |
| - [validation-hard15-surya.json](/Users/cdjk/github/llm/ocr-tech/configs/validation-hard15-surya.json) | |
| - [line-align hard15 validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-claimpack-v2-line-align-hard15/validation.md) | |
| - [k2 manifest-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-claimpack-v2-surya-hard15-k2/manifest-audit.md) | |
| - [table/figure dev pack manifest](/Users/cdjk/github/llm/ocr-tech/data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl) | |
| - [table/figure dev pack audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-eval-pack-table-figure-dev-v1-k2/eval-pack-audit.md) | |
| Diagnostic summary: | |
| | slice | sample_count | Surya CER | | |
| | --- | ---: | ---: | | |
| | figure | 20 | 0.253157 | | |
| | table | 20 | 0.252781 | | |
| | multi_page | 10 | 0.196472 | | |
| | scan | 10 | 0.194861 | | |
| | low_contrast | 10 | 0.193317 | | |
| | blur | 10 | 0.190940 | | |
| | clean | 10 | 0.182256 | | |
| Interpretation: | |
| - Surya's hardest absolute CER pockets on rendered-v2 are table-ledger and figure-caption samples, not generic blur/skew alone. | |
| - Tesseract is worse than Surya on `49` of `50` samples and better on only `eval-form-notice-003-blur`; that single oracle win is too small to justify more Surya/Tesseract routing work. | |
| - A ranked `15`-sample `hard_eval` manifest now exists for the next model branch; it contains `8` table samples and `7` figure samples, and its full file/reference audit passes on `k2`. | |
| - `configs/validation-hard15-surya.json` defines the pocket gate a future candidate must clear against Surya, including intersection gates for `hard_eval,table` and `hard_eval,figure`. | |
| - The current line-align candidate fails the hard15 gate: hard-eval CER is `0.482039` versus Surya `0.348706`, and hard table CER is `0.613636` versus Surya `0.363636`. | |
| - A separate `80`-sample image development pack now exists for table/figure work: `40` table samples, `40` figure samples, all five degradations. It has zero sample-id overlap and zero text-hash overlap with hard15, and its `k2` audit passes. | |
| - That development pack is not claim evidence; it is for model iteration before returning to the held-out hard15 and rendered-v2 gates. | |
| - The next credible recognizer branch should target table/figure-heavy mixed-script pages and preserve Surya's already-strong behavior on the other `49` samples. | |
| ### Table/Figure Development Pack - HF TrOCR Negative Result | |
| Generated artifact: | |
| - [split-summary.md](/Users/cdjk/github/llm/ocr-tech/data/splits-table-figure-dev-v1/split-summary.md) | |
| - [training_metrics.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-table-figure-dev-v1/training_metrics.json) | |
| - [model-card.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-table-figure-dev-v1/package/model-card.json) | |
| - [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-table-figure-dev-v1-model/model-audit.md) | |
| - [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-figure-dev-v1/summary.md) | |
| - [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-table-figure-dev-v1/benchmark/report.json) | |
| - [paddle_recognizer.yml](/Users/cdjk/github/llm/ocr-tech/runs/paddle-table-figure-dev-v1/paddle_recognizer.yml) | |
| Pack and split: | |
| - `80` image samples: `40` table and `40` figure | |
| - degradations: `clean`, `scan`, `blur`, `low_contrast`, `skew` | |
| - split: `60` train / `20` eval | |
| - grouping: `text_sha256` | |
| - text groups: `16` | |
| - split leakage: pass | |
| - train/eval manifest audits on `k2`: pass | |
| - hard15 overlap: `0` sample IDs and `0` text hashes | |
| Training result for `microsoft/trocr-small-printed` after one epoch: | |
| | metric | value | | |
| | --- | ---: | | |
| | eval_loss | 7.884517 | | |
| | eval_cer | 3.062119 | | |
| | eval_wer | 3.562689 | | |
| | eval_exact_line_accuracy | 0.000000 | | |
| Benchmark result on the `20`-sample dev eval: | |
| | system | CER | table_cell_f1 | reading_order_pair_accuracy | | |
| | --- | ---: | ---: | ---: | | |
| | HF table/figure candidate | 2.422001 | 0.000000 | 0.000000 | | |
| | Surya | 0.563399 | 0.858333 | 0.550000 | | |
| | Tesseract | 0.546227 | 0.000000 | 0.695000 | | |
| Interpretation: | |
| - The model package audits cleanly, so this is a quality failure rather than a packaging failure. | |
| - Paired CER was `0` wins, `20` losses, and `0` ties against Surya. | |
| - Paired CER was also `0` wins, `20` losses, and `0` ties against Tesseract. | |
| - Naive full-page TrOCR fine-tuning is not a serious next step for table/figure-heavy document pages. | |
| - The next model branch should use line/cell crops or a structure-aware page/document model rather than treating full document images as single recognition targets. | |
| ### Table/Figure Crop Development Pack - HF TrOCR Negative Result | |
| Generated artifact: | |
| - [crop-manifest-summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/crops-table-figure-dev-v1-k2/crop-manifest-summary.md) | |
| - [crop manifest audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/crops-table-figure-dev-v1-k2/manifest-audit.md) | |
| - [crop split-summary.md](/Users/cdjk/github/llm/ocr-tech/data/splits-crops-table-figure-dev-v1/split-summary.md) | |
| - [training_metrics.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-crops-table-figure-dev-v1/training_metrics.json) | |
| - [model-card.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-crops-table-figure-dev-v1/package/model-card.json) | |
| - [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-crops-table-figure-dev-v1-model/model-audit.md) | |
| - [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-crops-table-figure-dev-v1/summary.md) | |
| - [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-crops-table-figure-dev-v1/benchmark/report.json) | |
| Pack and split: | |
| - source rendered samples: `80` | |
| - extracted crops: `544` | |
| - crop types: `384` table cells, `128` lines, `32` figure captions | |
| - skipped candidates: `32` skew pages because pre-rotation bboxes are invalid | |
| - split: `480` train / `64` eval | |
| - grouping: `text_sha256` | |
| - text groups: `68` | |
| - crop manifest audit on `k2`: pass | |
| - split leakage: pass | |
| Training result for `microsoft/trocr-small-printed` after one crop-level epoch: | |
| | metric | value | | |
| | --- | ---: | | |
| | eval_loss | 3.087561 | | |
| | eval_cer | 0.959831 | | |
| | eval_wer | 1.363715 | | |
| | eval_exact_line_accuracy | 0.078125 | | |
| Benchmark result on the `64`-sample crop eval: | |
| | system | CER | | |
| | --- | ---: | | |
| | HF crop candidate | 1.043369 | | |
| | Surya | 0.475309 | | |
| | Tesseract | 0.077453 | | |
| Interpretation: | |
| - Crop-level training is a better problem shape than full-page recognition, but this recognizer is still not competitive. | |
| - Paired CER against Surya was `7` wins, `50` losses, and `7` ties. | |
| - Paired CER against Tesseract was `0` wins, `64` losses, and `0` ties. | |
| - Tesseract is extremely strong on these rendered crop images, so near-term gains are more likely from better crop/layout detection plus strong crop OCR than from this small synthetic TrOCR fine-tune. | |
| ### Hard15 - Table Cell Refinement Composite | |
| Generated artifact: | |
| - [unsafe model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-v1/model-card.json) | |
| - [unsafe summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-hard15-v1/summary.md) | |
| - [unsafe cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-always-hard15-v1/table-cell-analysis.md) | |
| - [safe model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-empty-v1/model-card.json) | |
| - [safe summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-empty-hard15-v1/summary.md) | |
| - [safe cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-empty-hard15-v1/table-cell-analysis.md) | |
| - [safe validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-hard15-v1/validation.md) | |
| - [safe+dedup model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-empty-dedup-v1/model-card.json) | |
| - [safe+dedup summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-empty-dedup-hard15-v3/summary.md) | |
| - [safe+dedup cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-empty-dedup-hard15-v3/table-cell-analysis.md) | |
| - [safe+dedup validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-dedup-hard15-v3/validation.md) | |
| Headline hard15 comparison: | |
| | system | hard_eval CER | table_cell_f1 | reading_order_pair_accuracy | | |
| | --- | ---: | ---: | ---: | | |
| | Surya baseline | 0.348706 | 0.979167 | 0.520000 | | |
| | Table-cell refine, `always` | 0.348706 | 0.593750 | 0.520000 | | |
| | Table-cell refine, `empty` | 0.348706 | 0.979167 | 0.520000 | | |
| | Table-cell refine, `empty` + parser de-dup | 0.348706 | 0.979167 | 0.520000 | | |
| Interpretation: | |
| - The unsafe table-cell crop replacement policy damaged Surya's already-good table cells. | |
| - The conservative empty-cell-only policy avoided regression but did not improve any hard15 metric. | |
| - The safe policy's hard15 validation status is `not_validated`. | |
| - The parser de-dup branch removes duplicate overlapping table exports for both candidate and Surya, so it improves artifact cleanliness but does not create a candidate advantage. | |
| - Exact crop OCR being strong is not enough; the deployment problem is reliable cell detection and knowing when a replacement is safer than the structure engine's existing cell text. | |
| Cell-level result: | |
| - Surya is already exact on `94/96` hard15 reference table cells. | |
| - Unsafe replacement is exact on only `57/96` cells. | |
| - Unsafe replacement improves `0` cells over Surya and worsens `37`. | |
| - Safe+dedup is exact on `94/96` cells, with `8` predicted tables for `8` reference tables and `0` duplicate overlapping tables on the v3 rerun. | |
| ### Hard15 - Figure Caption Refinement Composite | |
| Generated artifact: | |
| - [model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/figure-caption-refine-v1/model-card.json) | |
| - [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-figure-caption-refine-v1-model/model-audit.md) | |
| - [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-figure-caption-refine-hard15-v2/summary.md) | |
| - [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-figure-caption-refine-hard15-v2/benchmark/report.json) | |
| - [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-figure-caption-refine-hard15-v2/validation.md) | |
| - [labels model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/figure-caption-refine-labels-v1/model-card.json) | |
| - [labels summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-figure-caption-refine-labels-hard15-v1/summary.md) | |
| - [labels validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-figure-caption-refine-labels-hard15-v1/validation.md) | |
| Headline hard15 comparison after the parser caption-attachment fix: | |
| | system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 | | |
| | Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 | | |
| | Figure-caption refine | 0.228008 | 0.052478 | 1.000000 | 0.979167 | 0.520000 | | |
| | Figure-caption refine + labels | 0.204610 | 0.008746 | 1.000000 | 0.979167 | 0.520000 | | |
| Validation result: | |
| - basic branch overall status: `not_validated` | |
| - basic branch figure-caption gate: pass, `6` wins / `0` losses / `1` tie against Surya, improvement `0.039845` | |
| - label branch overall status: `not_validated` | |
| - label branch overall hard-eval CER gate: pass, improvement `0.032228` | |
| - label branch figure CER gate: pass, improvement `0.069061` | |
| - label branch figure-caption gate: pass, `7` wins / `0` losses / `0` ties against Surya, improvement `0.083576` | |
| - label branch table gates: fail, ties against Surya | |
| Interpretation: | |
| - This is a real slice win over Surya on figure-caption extraction. | |
| - It is not broad enough for a SOTA claim. | |
| - Tesseract has better caption text but worse figure detection/table structure; the composite keeps Surya's structure and selectively borrows caption text. | |
| - Safe empty-only replacement ties all `96` cells with Surya because there are no missing Surya cells in this pocket. | |
| - The only remaining Surya cell errors are `किरण` predicted as `करिण` on `eval-table-ledger-002-clean` and `eval-table-ledger-002-scan`. | |
| ### Rendered Blur-Only Expensive-Baseline Subset | |
| Generated artifact: | |
| - [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-blur5-table/paper-table.md) | |
| - [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-blur5-table/paper-table.json) | |
| Headline table: | |
| | system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | Candidate | 5 | 0.223273 | 1.000000 | 0.783636 | 3.784279 | | |
| | Surya | 5 | 0.197601 | 1.000000 | 0.750909 | 2.943875 | | |
| | Tesseract | 5 | 0.732364 | 0.000000 | 0.823636 | 0.220515 | | |
| | Stock Paddle | 5 | 2.810426 | 0.000000 | 1.000000 | 7.440482 | | |
| | GLM-OCR | 5 | 2.820269 | 0.000000 | 1.000000 | 16.846619 | | |
| | PaddleOCR-VL | 5 | 18.470663 | 0.000000 | 0.750909 | 32.793166 | | |
| Interpretation: | |
| - Candidate is materially stronger than Tesseract, stock-Paddle, GLM-OCR, and PaddleOCR-VL on this degraded rendered subset. | |
| - Candidate still does not beat raw Surya on text CER. | |
| - The external VLM-style baselines are also much slower than the current candidate here. | |
| ## Branch Chronology | |
| This is the shortest honest story of the project so far. | |
| | branch | eval setting | candidate CER | baseline CER | paired result | verdict | evidence | | |
| | --- | --- | ---: | ---: | --- | --- | --- | | |
| | HF TrOCR recognizer | mixed Nepali/English line-style holdout, 180 samples | 0.586139 | 0.329012 vs Tesseract | 36 wins, 97 losses, 47 ties | runnable but not competitive | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-hf-vs-tesseract-summary/summary.json) | | |
| | Tesseract + ByT5 corrector | same 180-sample mixed holdout | 6.912065 | 0.329012 vs Tesseract | 0 wins, 170 losses, 10 ties | failed badly | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-corrector-vs-tesseract-summary/summary.md) | | |
| | quality router (`psm=6` vs `psm=13`) | 91-sample mixed holdout | 0.340608 | 0.342234 vs Tesseract | 2 wins, 1 loss, 88 ties | tiny gain, plausible baseline only | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-quality-router-holdout/summary.json) | | |
| | ranked PSM ensemble (`6/7/11/13`) | same 91-sample mixed holdout | 0.358352 | 0.342234 vs Tesseract | 9 wins, 16 losses, 66 ties | overfit, worse than baseline | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-psm-ensemble-holdout/summary.json) | | |
| | script router (`tesseract -> surya`) | same 91-sample mixed holdout | 0.313084 | 0.342234 vs Tesseract | 9 wins, 0 losses, 82 ties | strongest older line-style result | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-script-router-holdout/summary.json) | | |
| | script router + structure backfill | rendered claim-ready document pack, 25 samples | 0.248654 | 0.689214 vs Tesseract | 9 wins, 1 loss, 15 ties | strong document-pack improvement over Tesseract | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-rendered-backfill/summary.json) | | |
| | raw Surya on rendered claim pack | rendered claim-ready document pack, 25 samples | 0.195260 | 0.248654 vs Candidate | candidate loses on text CER | current text leader on rendered pack | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-surya-compare/summary.json) | | |
| | rendered-v2 script router | rendered claim-ready document holdout, 50 samples | 0.180893 | 0.180893 vs Surya | exact CER tie, worse table structure | routed candidate collapses to Surya on calibration; not a meaningful new engine | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-script-router/summary.json) | | |
| | rendered-v2 Surya-primary Latin router | rendered claim-ready document holdout, 50 samples | 0.416854 | 0.180893 vs Surya | 0 wins, 12 losses, 38 ties on CER | failed badly; inverted script routing is not a viable answer | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-latin-router/summary.json) | | |
| | rendered-v2 Surya quality router | rendered claim-ready document holdout, 50 samples | 0.181128 | 0.180893 vs Surya | 0 wins, 1 loss, 49 ties on CER | nearly indistinguishable from Surya and still not better | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-quality-router/summary.json) | | |
| | rendered-v2 line-align composite | rendered claim-ready document holdout, 50 samples | 0.310549 | 0.180893 vs Surya | 0 wins, 7 losses, 43 ties on CER | better reading order, but text and table regressions are too large | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-line-align-summary/summary.json) | | |
| | rendered-v2 Surya/Tesseract sample oracle | rendered claim-ready document holdout, 50 samples | 0.180529 | 0.180893 vs Surya | oracle picks Tesseract only 1/50 samples | upper bound shows routing headroom is essentially exhausted | [oracle-analysis.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-analysis.json) | | |
| | HF TrOCR table/figure dev | table/figure development eval, 20 samples | 2.422001 | 0.563399 vs Surya; 0.546227 vs Tesseract | 0 wins, 20 losses, 0 ties vs both | failed badly; full-page TrOCR fine-tuning is not viable here | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-figure-dev-v1/summary.md) | | |
| | HF TrOCR crop dev | table/figure crop eval, 64 samples | 1.043369 | 0.475309 vs Surya; 0.077453 vs Tesseract | 7 wins, 50 losses, 7 ties vs Surya; 0 wins, 64 losses vs Tesseract | crop shape helps but this recognizer still fails | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-crops-table-figure-dev-v1/summary.md) | | |
| | table-cell refine, unsafe | hard15 rendered-v2 pocket, 15 samples | 0.348706 | 0.348706 vs Surya | text tie, table F1 regresses from 0.979167 to 0.593750 | replacement policy is unsafe | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-hard15-v1/summary.md) | | |
| | table-cell refine, empty-only | hard15 rendered-v2 pocket, 15 samples | 0.348706 | 0.348706 vs Surya | exact tie across CER/table/reading order; validation not_validated | safe but no improvement | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-hard15-v1/validation.md) | | |
| | normalized script router, hard-timeout rerun | normalized mixed Nepali/English line-style eval, 180 samples | 0.354840 | 0.329012 vs Tesseract; 3.157317 vs Surya on 178 scored Surya rows | 17 wins, 4 losses, 159 ties on CER vs Tesseract but worse aggregate mean; 2 Surya rows timed out at 45s | not a better engine; benchmark watchdog works and routing still loses overall CER | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-mixed-v1-normalized-script-router-hard-timeout/outputs/summary-mixed-v1-normalized-script-router-hard-timeout/summary.md) | | |
| | PaddleOCR recognizer hard-50 failure slice | top 50 failures from AWS G6 trained PaddleOCR recognizer, remapped to normalized k2 eval paths | n/a | n/a | 50 failures: 38 English line-crop synthetic, 12 Nepali synthetic; k2 manifest audit passed | next recognizer gate; do not train on it | [manifest-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-paddle-mixed-v2-hard50-normalized/outputs/manifest-audit-paddle-mixed-v2-cuda-hard50-normalized/manifest-audit.md) | | |
| | PaddleOCR hard-50 gate negative control | failed AWS G6 trained PaddleOCR recognizer evaluated on the hard-50 gate | 0.887557 | 0.080249 vs Tesseract | fails all four hard gates; English line-crop CER `0.852048` vs `0.031244` | validates the gate catches the known recognizer collapse | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-paddle-mixed-v2-cuda-hard50-tesseract-negative/validation.md) | | |
| | line-level Tesseract/Paddle composite | hard-50 failure slice, Tesseract primary with PaddleOCR secondary only for Devanagari-looking lines | 0.276489 | 0.120106 vs Tesseract | English line-crop ties exactly (`0.060429` CER); Nepali collapses (`0.960679` vs `0.309083` CER, `1.000000` vs `0.711607` WER) | safe English preservation but failed Devanagari override; do not promote this recognizer | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-line-script-paddle-hard50-v2/validation.md) | | |
| | PaddleOCR v3 CUDA recognizer | G6-trained lower-LR PaddleOCR branch on hard-50 admission gate | 0.921416 | 0.080249 vs Tesseract | 0 wins, 50 losses on CER; English line-crop CER `0.917511` vs `0.031244`; Nepali WER `1.055556` vs `0.716369` | failed worse than v2; stop branch before full mixed eval | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/aws-g6-paddle-mixed-v3-cuda/outputs/validation-paddle-mixed-v3-cuda-hard50-rescored/validation.md) | | |
| | PaddleOCR v4 CUDA recognizer | G6-trained Devanagari PP-OCRv3 branch, LR `0.00005`, 20 epochs, evaluated on hard-50 admission gate | 0.940500 | 0.080249 vs Tesseract | 0 wins, 50 losses on CER; English line-crop CER `0.950730` vs `0.031244`; Nepali CER `0.908108` vs `0.235433` | failed worse than v2/v3; the same PaddleOCR mixed-script fine-tune branch should not be the engine core | [artifact bundle](/Users/cdjk/github/llm/ocr-tech/outputs/aws-g6-paddle-mixed-v4-cuda/ocrtech-paddle-mixed-v4-cuda-artifacts.tgz) | | |
| ## Current Best-Evidence Statement | |
| The strongest evidence-backed statement available today is: | |
| 1. The current Tesseract-plus-Surya backfill candidate is a real document-OCR improvement over Tesseract and several weaker baselines on internal rendered Nepali/English document packs. | |
| 2. On the larger rendered-v2 holdout, every Tesseract/Surya composite tested so far either collapses to Surya, trails Surya, becomes effectively indistinguishable from Surya, or trades too much OCR accuracy for modest ordering gains. | |
| 3. On the cleaned 180-sample mixed line-style eval with hard per-sample timeouts, the older script-router candidate also loses to Tesseract overall (`0.354840` vs `0.329012` CER), despite a small overall WER improvement (`0.469944` vs `0.486485`) and a Nepali WER improvement. | |
| 4. The trained PaddleOCR recognizer failure profile is now materialized as a normalized hard-50 eval gate: all top failures are synthetic line crops, mostly English (`38/50`), so the next recognizer branch must prove English preservation instead of only improving Nepali. | |
| 5. The line-level Tesseract/Paddle composite proves the candidate shape can preserve English while routing Devanagari-looking lines, but the available trained PaddleOCR recognizer is too weak on the Nepali hard slice to use as an override (`0.960679` vs Tesseract `0.309083` CER). | |
| 6. The second AWS G6 PaddleOCR attempt fixed recipe/runtime issues but failed the hard-50 gate even worse than the first (`0.921416` vs Tesseract `0.080249` CER), so this PP-OCRv3 Devanagari recipe family is not the engine path. | |
| 7. A sample-level oracle over raw Surya and raw Tesseract has almost no text-OCR headroom over Surya (`0.180529` vs `0.180893` CER), so more routing in this component family is not a serious path to SOTA. | |
| 8. The repo now has enough validation discipline and artifact traceability to support a serious comparative paper, but not a SOTA claim. | |
| ## Regeneration Workflow | |
| Rebuild the generated comparison tables with: | |
| ```bash | |
| ocrtech summarize-benchmark \ | |
| --benchmark-report outputs/live-k2-claimpack-rendered-backfill/report.json \ | |
| --out outputs/paper-claimpack-summary-tesseract \ | |
| --candidate candidate \ | |
| --baseline tesseract \ | |
| --metric cer | |
| ``` | |
| ```bash | |
| ocrtech summarize-benchmark \ | |
| --benchmark-report outputs/live-k2-claimpack-surya-compare/report.json \ | |
| --out outputs/paper-claimpack-summary-surya \ | |
| --candidate candidate \ | |
| --baseline surya \ | |
| --metric cer | |
| ``` | |
| ```bash | |
| ocrtech summarize-benchmark \ | |
| --benchmark-report outputs/live-k2-claimpack-stock-paddle-fixed/report.json \ | |
| --out outputs/paper-claimpack-summary-stock-paddle \ | |
| --candidate candidate \ | |
| --baseline stock-paddle \ | |
| --metric cer | |
| ``` | |
| ```bash | |
| ocrtech paper-benchmark-table \ | |
| --summary outputs/paper-claimpack-summary-tesseract/summary.json \ | |
| --summary outputs/paper-claimpack-summary-surya/summary.json \ | |
| --summary outputs/paper-claimpack-summary-stock-paddle/summary.json \ | |
| --out outputs/paper-claimpack-table \ | |
| --system candidate \ | |
| --system surya \ | |
| --system tesseract \ | |
| --system stock-paddle \ | |
| --label 'candidate=Candidate' \ | |
| --label 'surya=Surya' \ | |
| --label 'tesseract=Tesseract' \ | |
| --label 'stock-paddle=Stock Paddle' \ | |
| --source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' \ | |
| --source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' \ | |
| --source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' \ | |
| --source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' \ | |
| --metric cer \ | |
| --metric table_cell_f1 \ | |
| --metric reading_order_pair_accuracy \ | |
| --metric latency_seconds | |
| ``` | |
| ```bash | |
| ocrtech summarize-benchmark \ | |
| --benchmark-report outputs/live-k2-rendered-blur5/report.json \ | |
| --out outputs/paper-blur5-summary \ | |
| --candidate candidate \ | |
| --baseline tesseract \ | |
| --baseline surya \ | |
| --baseline stock-paddle \ | |
| --baseline glm-ocr \ | |
| --baseline paddleocr-vl \ | |
| --metric cer | |
| ``` | |
| ```bash | |
| ocrtech paper-benchmark-table \ | |
| --summary outputs/paper-blur5-summary/summary.json \ | |
| --out outputs/paper-blur5-table \ | |
| --system candidate \ | |
| --system surya \ | |
| --system tesseract \ | |
| --system stock-paddle \ | |
| --system glm-ocr \ | |
| --system paddleocr-vl \ | |
| --label 'candidate=Candidate' \ | |
| --label 'surya=Surya' \ | |
| --label 'tesseract=Tesseract' \ | |
| --label 'stock-paddle=Stock Paddle' \ | |
| --label 'glm-ocr=GLM-OCR' \ | |
| --label 'paddleocr-vl=PaddleOCR-VL' \ | |
| --metric cer \ | |
| --metric table_cell_f1 \ | |
| --metric reading_order_pair_accuracy \ | |
| --metric latency_seconds | |
| ``` | |
| ```bash | |
| ocrtech paper-benchmark-table \ | |
| --summary outputs/live-k2-claimpack-v2-script-router/summary.json \ | |
| --out outputs/paper-claimpack-v2-table \ | |
| --system candidate \ | |
| --system surya \ | |
| --system tesseract \ | |
| --label 'candidate=Candidate' \ | |
| --label 'surya=Surya' \ | |
| --label 'tesseract=Tesseract' \ | |
| --metric cer \ | |
| --metric table_cell_f1 \ | |
| --metric reading_order_pair_accuracy \ | |
| --title 'Rendered Claim Pack V2 Holdout Comparison' | |
| ``` | |
| ```bash | |
| ocrtech paper-benchmark-table \ | |
| --summary outputs/live-k2-claimpack-v2-surya-quality-router/summary.json \ | |
| --out outputs/paper-claimpack-v2-surya-quality-router-table \ | |
| --system candidate \ | |
| --system surya \ | |
| --system tesseract \ | |
| --label 'candidate=Candidate' \ | |
| --label 'surya=Surya' \ | |
| --label 'tesseract=Tesseract' \ | |
| --metric cer \ | |
| --metric table_cell_f1 \ | |
| --metric reading_order_pair_accuracy \ | |
| --metric latency_seconds \ | |
| --title 'Rendered Claim Pack V2 Holdout - Surya Quality Router' | |
| ``` | |
| ```bash | |
| ocrtech paper-benchmark-table \ | |
| --summary outputs/live-k2-claimpack-v2-line-align-summary/summary.json \ | |
| --out outputs/paper-claimpack-v2-line-align-table \ | |
| --system candidate \ | |
| --system surya \ | |
| --system tesseract \ | |
| --label 'candidate=Candidate' \ | |
| --label 'surya=Surya' \ | |
| --label 'tesseract=Tesseract' \ | |
| --metric cer \ | |
| --metric table_cell_f1 \ | |
| --metric reading_order_pair_accuracy \ | |
| --metric latency_seconds \ | |
| --title 'Rendered Claim Pack V2 Holdout - Line Align Composite' | |
| ``` | |
| ```bash | |
| ocrtech benchmark-oracle \ | |
| --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \ | |
| --out outputs/paper-claimpack-v2-surya-tesseract-oracle \ | |
| --system surya \ | |
| --system tesseract \ | |
| --metric cer \ | |
| --oracle-name oracle-surya-tesseract | |
| ocrtech paper-benchmark-table \ | |
| --summary outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-summary/summary.json \ | |
| --out outputs/paper-claimpack-v2-oracle-table \ | |
| --system oracle-surya-tesseract \ | |
| --system surya \ | |
| --system tesseract \ | |
| --label 'oracle-surya-tesseract=Oracle Surya/Tesseract' \ | |
| --label 'surya=Surya' \ | |
| --label 'tesseract=Tesseract' \ | |
| --metric cer \ | |
| --metric table_cell_f1 \ | |
| --metric reading_order_pair_accuracy \ | |
| --metric latency_seconds \ | |
| --title 'Rendered Claim Pack V2 Holdout - Sample-Level Oracle' | |
| ``` | |
| ```bash | |
| ocrtech benchmark-error-analysis \ | |
| --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \ | |
| --out outputs/paper-claimpack-v2-surya-error-analysis \ | |
| --system surya \ | |
| --metric cer \ | |
| --top-n 15 | |
| ocrtech benchmark-error-analysis \ | |
| --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \ | |
| --out outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis \ | |
| --system surya \ | |
| --baseline tesseract \ | |
| --metric cer \ | |
| --top-n 15 | |
| ocrtech benchmark-error-analysis \ | |
| --benchmark-report outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-report.json \ | |
| --out outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis \ | |
| --system surya \ | |
| --baseline oracle-surya-tesseract \ | |
| --metric cer \ | |
| --top-n 10 | |
| ocrtech export-failure-manifest \ | |
| --source-manifest data/splits-claimpack-rendered-v2/eval.jsonl \ | |
| --top-failures outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json \ | |
| --out data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \ | |
| --split hard_eval | |
| ocrtech validate-claim \ | |
| --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \ | |
| --config configs/validation-hard15-surya.json \ | |
| --eval-manifest data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \ | |
| --out outputs/validation-claimpack-v2-line-align-hard15 | |
| ocrtech create-eval-pack \ | |
| --out data/eval-pack-table-figure-dev-v1 \ | |
| --input-format image \ | |
| --template table-ledger \ | |
| --template figure-caption \ | |
| --count-per-template 8 \ | |
| --variant-offset 100 \ | |
| --degradation clean \ | |
| --degradation scan \ | |
| --degradation blur \ | |
| --degradation low_contrast \ | |
| --degradation skew | |
| ``` | |
| Memory still is not part of these generated tables. The current benchmark artifact format does not record it. | |
| ## Hard15 Figure-Caption + Table-Lexicon Candidate | |
| Latest artifact set: | |
| - `outputs/figure-caption-table-lexicon-v1/model-card.json` | |
| - `outputs/experiment-figure-caption-table-lexicon-hard15-v3/benchmark/report.json` | |
| - `outputs/summary-figure-caption-table-lexicon-hard15-v3/summary.md` | |
| - `outputs/table-cell-analysis-figure-caption-table-lexicon-hard15-v3/table-cell-analysis.md` | |
| - `outputs/validation-figure-caption-table-lexicon-hard15-v3/validation.md` | |
| Headline table for internal hard15 only: | |
| | system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | candidate | 0.010671 | 0.008746 | 1.000000 | 1.000000 | 0.520000 | | |
| | Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 | | |
| | Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 | | |
| Validation status is still `not_validated`. | |
| Passed configured Surya gates: | |
| - overall hard_eval CER: `15` wins / `0` losses / `0` ties | |
| - table-slice CER: `8` wins / `0` losses / `0` ties | |
| - figure-slice CER: `7` wins / `0` losses / `0` ties | |
| - figure-caption CER: `7` wins / `0` losses / `0` ties | |
| Remaining configured failure: | |
| - table-cell F1: candidate `1.000000`, Surya `0.979167`, but bootstrap CI low is `0.000000` because there are only 2 improving table pages and 6 ties | |
| Cell-level table evidence: | |
| | system | reference_cells | exact_cells | exact_rate | mean_cer | | |
| | --- | ---: | ---: | ---: | ---: | | |
| | candidate | 96 | 96 | 1.000000 | 0.000000 | | |
| | Surya | 96 | 94 | 0.979167 | 0.010417 | | |
| | Tesseract | 96 | 0 | 0.000000 | 1.000000 | | |
| Use this result as an internal milestone, not a paper claim of broad SOTA. The candidate is lexicon-assisted for table names and uses guarded structural-title corrections (`वार्षिक प्रतिवेदन`, `दैनिक हिसाब`, `चित्र`, `निष्कर्ष`). A publishable table claim needs a larger independent table pack so confidence intervals are meaningful. | |
| ## Targeted Table-Ledger Validation Pack V2 | |
| Artifacts: | |
| - `data/eval-pack-table-ledger-claim-v1/manifests/ocrtech-eval-pack.jsonl` | |
| - `outputs/audit-table-ledger-claim-v1/eval-pack-audit.md` | |
| - `outputs/experiment-table-ledger-claim-v2/benchmark/report.json` | |
| - `outputs/summary-table-ledger-claim-v2/summary.md` | |
| - `outputs/table-cell-analysis-table-ledger-claim-v2/table-cell-analysis.md` | |
| - `outputs/validation-table-ledger-claim-v2b/validation.md` | |
| - `configs/validation-table-ledger-surya.json` | |
| Pack: | |
| - template: `table-ledger` | |
| - samples: `60` | |
| - cells: `720` | |
| - degradations: clean, scan, blur, low_contrast, skew | |
| - variant offset: `300` | |
| - claim-ready broad coverage: no, targeted table-only pack | |
| Validation status: `validated`. | |
| | metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low | | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | |
| | table_cell_f1 | 60 | 15 | 0 | 45 | 1.000000 | 0.979167 | 0.020833 | 0.012500 | | |
| | table_exact_match | 60 | 15 | 0 | 45 | 1.000000 | 0.750000 | 0.250000 | 0.150000 | | |
| | table_structure_similarity | 60 | 0 | 0 | 60 | 1.000000 | 1.000000 | 0.000000 | 0.000000 | | |
| Cell exactness: | |
| | system | reference_cells | exact_cells | exact_rate | mean_cell_cer | | |
| | --- | ---: | ---: | ---: | ---: | | |
| | candidate | 720 | 720 | 1.000000 | 0.000000 | | |
| | Surya | 720 | 705 | 0.979167 | 0.005208 | | |
| | Tesseract | 720 | 0 | 0.000000 | 0.999868 | | |
| Runtime on k2: | |
| | system | mean latency seconds | median latency seconds | | |
| | --- | ---: | ---: | | |
| | candidate | 3.017929 | 2.967943 | | |
| | Surya | 2.478481 | 2.453125 | | |
| | Tesseract | 0.146866 | 0.134735 | | |
| Paper-safe claim text: | |
| > On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved structured table extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, and passed bootstrap-gated validation for both table-cell F1 and table exact match. | |
| More precise paper-safe wording: | |
| > On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved table cell-content extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, while matching Surya's perfect grid-structure score. | |
| Do not generalize this to all tables. The pack has no merged cells, no nested headers, and no real scanned government tables. The scoring path now supports span-aware structured references, but this pack does not exercise that capability and the metric is still not true HTML-tree TEDS. | |
| ## Targeted Complex-Table Validation Pack V1 | |
| Artifacts: | |
| - `data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl` | |
| - `outputs/audit-complex-table-v1/eval-pack-audit.md` | |
| - `outputs/experiment-complex-table-v1/benchmark/report.json` | |
| - `outputs/summary-complex-table-v1/summary.md` | |
| - `outputs/table-cell-analysis-complex-table-v1/table-cell-analysis.md` | |
| - `outputs/validation-complex-table-v1/validation.md` | |
| - `configs/validation-complex-table-surya.json` | |
| Pack: | |
| - template: `complex-table-ledger` | |
| - samples: `40` | |
| - reference cells: `600` | |
| - span reference: merged header with `colspan=2` | |
| - degradations: clean, scan, blur, low_contrast | |
| - broad claim-ready coverage: no | |
| Validation status: `validated` for cell-content improvement plus structure non-regression. | |
| | metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low | | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | |
| | table_cell_f1 | 40 | 8 | 0 | 32 | 0.928333 | 0.915000 | 0.013333 | 0.005000 | | |
| | table_exact_match | 40 | 0 | 0 | 40 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | | |
| | table_structure_similarity | 40 | 0 | 0 | 40 | 0.875000 | 0.875000 | 0.000000 | 0.000000 | | |
| Cell exactness: | |
| | system | reference_cells | exact_cells | exact_rate | mean_cell_cer | | |
| | --- | ---: | ---: | ---: | ---: | | |
| | candidate | 600 | 557 | 0.928333 | 0.067667 | | |
| | Surya | 600 | 549 | 0.915000 | 0.071000 | | |
| | Tesseract | 600 | 40 | 0.066667 | 0.933333 | | |
| Paper-safe claim text: | |
| > On a targeted generated complex-table pack with span-aware references, the lexicon-assisted candidate improved table cell-content F1 over Surya but did not improve merged-header reconstruction; both systems failed exact table match and tied on span-aware structure similarity. | |
| This is a negative result for complex table structure. The next model/parser branch must preserve or infer rowspans/colspans before we can claim complex-table document understanding. | |
| ## Targeted Complex-Table Span-Inference Pack V1 | |
| After adding conservative repeated-header colspan inference, we reran the same `40`-sample generated complex-table pack on k2. | |
| Artifacts: | |
| - `outputs/experiment-complex-table-span-infer-v1/benchmark/report.json` | |
| - `outputs/experiment-complex-table-span-infer-v1/validation/validation.md` | |
| - `outputs/summary-complex-table-span-infer-v1/summary.md` | |
| - `outputs/table-cell-analysis-complex-table-span-infer-v1/table-cell-analysis.md` | |
| Validation status: `validated`. | |
| | metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low | | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | |
| | table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 | | |
| | table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 | | |
| | table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 | | |
| Cell exactness: | |
| | system | reference_cells | exact_cells | exact_rate | mean_cell_cer | | |
| | --- | ---: | ---: | ---: | ---: | | |
| | candidate | 600 | 597 | 0.995000 | 0.001000 | | |
| | Surya | 600 | 549 | 0.915000 | 0.071000 | | |
| | Tesseract | 600 | 40 | 0.066667 | 0.933333 | | |
| Paper-safe claim text: | |
| > On a targeted generated complex-table pack with one repeated merged-header pattern, a conservative span-inference postprocessor improved table structure extraction over Surya, raising span-aware structure similarity from 0.875 to 1.000 and table exact match from 0.000 to 0.925. | |
| Do not generalize this to all complex tables. This is generated data with a single span pattern and a heuristic parser repair. The next paper-grade check needs real scanned tables with varied merged cells, rowspans, nested headers, and a true HTML/TEDS scorer. | |
| ## Targeted Complex-Table TEDS-Style Metrics V1 | |
| We added deterministic ordered table-tree edit metrics and reran the same complex-table pack with these gates included. | |
| Artifacts: | |
| - `outputs/experiment-complex-table-teds-v1/benchmark/report.json` | |
| - `outputs/experiment-complex-table-teds-v1/validation/validation.md` | |
| - `outputs/summary-complex-table-teds-v1/summary.md` | |
| - `outputs/table-cell-analysis-complex-table-teds-v1/table-cell-analysis.md` | |
| Validation status: `validated`. | |
| | metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low | | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | |
| | table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 | | |
| | table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 | | |
| | table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 | | |
| | table_teds_structure | 40 | 40 | 0 | 0 | 1.000000 | 0.904762 | 0.095238 | 0.095238 | | |
| | table_teds_text | 40 | 40 | 0 | 0 | 0.996250 | 0.891667 | 0.104583 | 0.098690 | | |
| Paper-safe claim text: | |
| > On a targeted generated complex-table pack, the candidate improved ordered table-tree metrics over Surya, reaching 1.000 structure-only TEDS-style similarity and 0.996 text-sensitive TEDS-style similarity, compared with Surya's 0.905 and 0.892 respectively. | |
| This is still not a public-benchmark SOTA claim. The metrics are deterministic and closer to table-structure scoring, but the pack is generated and covers one repeated-header merged-cell pattern. | |
| ## HF Recognizer Next V1 Admission Benchmark | |
| The AWS G6 trained HF recognizer was evaluated on the `recognizer-next-v1` | |
| held-out eval manifest against Tesseract on `k2`. | |
| Artifacts: | |
| - `outputs/k2-hf-recognizer-next-v1-full-cpu/benchmark/report.json` | |
| - `outputs/k2-hf-recognizer-next-v1-full-cpu/summary/summary.md` | |
| - candidate model card: | |
| `outputs/aws-g6-hf-recognizer-next-v1/package/model-card-cpu.json` | |
| Run status: | |
| - candidate rows: `244` ok, `0` errors | |
| - Tesseract rows: `244` ok, `0` errors | |
| - paired CER rows with text references: `180` | |
| | slice | pairs | candidate CER | Tesseract CER | improvement | wins | losses | ties | | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | |
| | all | 180 | 0.226628 | 0.329012 | 0.102384 | 74 | 54 | 52 | | |
| | english/en/line_crop | 150 | 0.145190 | 0.243379 | 0.098189 | 56 | 42 | 52 | | |
| | nepali | 30 | 0.633814 | 0.757175 | 0.123360 | 18 | 12 | 0 | | |
| | real | 15 | 0.832255 | 1.162917 | 0.330662 | 12 | 3 | 0 | | |
| | synthetic | 165 | 0.171571 | 0.253202 | 0.081631 | 62 | 51 | 52 | | |
| Paper-safe claim text: | |
| > On the held-out `recognizer-next-v1` text-reference slices, the trained HF | |
| > recognizer improves paired CER over Tesseract, reducing aggregate CER from | |
| > 0.329 to 0.227 across 180 paired samples. | |
| Do not call this SOTA. The result is a recognizer admission benchmark against | |
| Tesseract only; the branch still needs Surya, stock PaddleOCR, GLM-OCR, public | |
| document benchmarks, table structure metrics, reading-order validation, and | |
| document reconstruction gates. | |