|
Download ocr-tech/docs/EXPERIMENTS.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 61.5 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/EXPERIMENTS.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/EXPERIMENTS.md
-
curl -L -o EXPERIMENTS.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/EXPERIMENTS.md
61.5 kB
# Experiment Workflow
If you are writing a paper from these runs, read [PAPER_NOTES.md](PAPER_NOTES.md) alongside this file. This document is the raw chronology; `PAPER_NOTES.md` is the stricter claim ledger.
`run-experiment` is the reproducible wrapper around benchmark and validation. It records the manifest hash, train manifest hashes, environment metadata, baseline list, benchmark report path, and validation status.
## Smoke Experiment
```bash
ocrtech create-eval-pack --out data/eval-pack-smoke --input-format text
ocrtech run-experiment \
--eval-manifest data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack-smoke/references \
--baselines ours \
--skip-validation \
--out outputs/smoke-experiment
```
This proves the pipeline and scoring path works. It is not OCR evidence.
## Claim Experiment
Run preflight first:
```bash
ocrtech preflight-claim \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--candidate-model-config runs/recognizer/package/model-card.json \
--validation-config validation.json \
--train-manifest data/splits/train.jsonl \
--out outputs/preflight
```
If the candidate is a routed composite, do not under-specify the baseline list. A claim-grade `script_select_composite` run must include its raw components in the same experiment, for example `candidate,tesseract,surya`, otherwise `preflight-claim` now fails `candidate-components`.
```bash
ocrtech run-experiment \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--candidate-model-config runs/recognizer/package/model-card.json \
--validation-config validation.json \
--train-manifest data/splits/train.jsonl \
--preflight \
--require-trained-recognizer \
--require-model-admission \
--out outputs/claim-experiment \
--require-validated
```
`--preflight` runs the same claim-readiness checks inside `run-experiment` and
stops before benchmark output is produced if the setup cannot satisfy the
configured gates. `--require-validated` returns a non-zero exit code unless
validation status is `validated`.
After the internal claim experiment and the public suites are all complete, run `review-claim`; see [CLAIM_REVIEW.md](CLAIM_REVIEW.md).
Use `summarize-benchmark` when you need aggregate and paired slice comparisons from a raw `benchmark/report.json`:
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/claim-experiment/benchmark/report.json \
--out outputs/claim-experiment/benchmark-summary \
--candidate candidate \
--baseline tesseract \
--metric cer
```
Use `paper-benchmark-table` when you need one paper-facing table merged from multiple `summary.json` artifacts, for example separate candidate-vs-Tesseract, candidate-vs-Surya, and candidate-vs-stock-Paddle summaries from the same pack:
```bash
ocrtech paper-benchmark-table \
--summary outputs/paper-claimpack-summary-tesseract/summary.json \
--summary outputs/paper-claimpack-summary-surya/summary.json \
--summary outputs/paper-claimpack-summary-stock-paddle/summary.json \
--out outputs/paper-claimpack-table \
--system candidate \
--system surya \
--system tesseract \
--system stock-paddle \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--label 'stock-paddle=Stock Paddle' \
--source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' \
--source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' \
--source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' \
--source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds
```
The `--source-preference` pins each system to one summary artifact. That matters when the same candidate was rerun in separate comparisons and latency differs across those reruns.
## Outputs
```text
outputs/claim-experiment/
benchmark/report.json
benchmark/report.md
validation/validation.json
validation/validation.md
experiment.json
experiment.md
```
`experiment.json` is the artifact to attach to a claim review. It does not replace the full benchmark and validation reports.
For candidate model-card requirements, see [MODELS.md](MODELS.md).
## Verified k2 HF Runs
As of June 14, 2026, the most useful live experiment series on `k2` is a candidate-only TrOCR comparison over one held-out mixed Nepali/English eval manifest with `180` samples. This is workflow evidence, not claim evidence, because it does not yet include Tesseract, stock Paddle, GLM-OCR, PaddleOCR-VL, or public benchmark suites.
Corpus construction used:
- `97` valid `gauravgiri` Nepali real rows from the first `100`
- `100` `prashant0919` Nepali synthetic rows
- `1000` NVIDIA English synthetic line crops from `nvidia/OCR-Synthetic-Multilingual-v1 --hf-subset en --split test`
- deterministic split: `1017` train / `180` eval
- materialized references from the eval manifest
- rebalanced training variant: `english=850`, `nepali=850`
Artifacts pulled from `k2` into this workspace:
- `outputs/live-k2-baseline-bundle/outputs/experiment-hf-smoke-on-mixed-v1/benchmark/report.json`
- `outputs/live-k2-baseline-bundle/outputs/experiment-hf-mixed-v1/benchmark/report.json`
- `outputs/live-k2-rebalanced-bundle/outputs/experiment-hf-mixed-v1-rebalanced/benchmark/report.json`
- `outputs/live-k2-rebalanced-bundle/runs/hf-recognizer-mixed-v1-rebalanced/training_metrics.json`
- `outputs/live-k2-rebalanced-bundle/data/splits-mixed-v1/train-rebalanced-nep-eng-rebalance.json`
- `outputs/live-k2-vs-tesseract-bundle/outputs/experiment-hf-mixed-v1-rebalanced-vs-tesseract/benchmark/report.json`
Benchmark means on that same eval split:
| Run | Overall CER | Overall WER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: | ---: |
| smoke | 11.613341 | 2.637120 | 9.191231 | 23.723889 | 19.593537 |
| mixed | 1.628157 | 2.916153 | 0.379729 | 7.870295 | 12.702701 |
| rebalanced | 0.586139 | 0.801844 | 0.300454 | 2.014560 | 1.770016 |
The rebalanced run is the first one here that materially improves both English and Nepali on the same held-out split instead of trading one off against the other. It is still not a SOTA claim because:
- it is candidate-only
- it uses one internal eval manifest, not the full internal document pack plus public suites
- it has no paired validation against named baselines yet
- it does not measure document-structure leadership, only OCR metrics from the current report path
That last caveat is no longer theoretical. The first named baseline check on the same eval split is already negative for the project claim:
| Baseline | Overall CER | Overall WER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: | ---: |
| candidate | 0.586139 | 0.801844 | 0.300454 | 2.014560 | 1.770016 |
| tesseract | 0.329012 | 0.486485 | 0.243379 | 0.757175 | 1.162917 |
So the current state is precise:
- we fixed our own mixed-script HF candidate substantially
- we have a reproducible training and benchmark path on `k2`
- we still do not beat Tesseract on this held-out eval
We also tested a mined-correction fallback instead of pretending post-OCR cleanup would obviously help. A ByT5 corrector was trained on `1700` Tesseract noisy/clean pairs mined from the rebalanced training manifest, then packaged as a `text_correction_composite` candidate with raw Tesseract as the base engine. That result was much worse than raw Tesseract:
| Baseline | Overall CER | Overall WER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: | ---: |
| Tesseract + ByT5 corrector | 6.912065 | 3.312896 | 6.578699 | 8.578897 | 15.254935 |
| raw tesseract | 0.329012 | 0.486485 | 0.243379 | 0.757175 | 1.162917 |
Paired CER on the same `180` held-out samples was decisive: `0` wins, `170` losses, `10` ties. The corrector trainer metrics were already warning us that this was not a good model (`eval_cer=7.067461`, `eval_wer=3.334917`, `eval_exact_line_accuracy=0.0`), and the benchmark confirmed it.
Artifacts recorded locally:
- `outputs/live-k2-corrector-vs-tesseract-summary/summary.json`
- `outputs/live-k2-corrector-vs-tesseract-summary/summary.md`
We also tested whether Tesseract itself had enough headroom to matter before touching the recognizer again. Three direct variant checks on the same held-out `180`-sample eval all lost overall to raw default Tesseract (`psm=6`):
| Candidate | Overall CER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: |
| raw tesseract (`psm=6`) | 0.329012 | 0.243379 | 0.757175 | 1.162917 |
| tesseract (`psm=7`) | 0.353708 | 0.242640 | 0.909049 | 1.104155 |
| tesseract (`psm=11`) | 0.344319 | 0.258868 | 0.771572 | 1.073521 |
| tesseract (`psm=13`) | 0.364878 | 0.243850 | 0.970017 | 1.004421 |
That said, `psm=13` was not uniformly useless. Using the existing `quality_score` as a routing heuristic, a `quality_select_composite` candidate that keeps default Tesseract unless `psm=13` is better by more than `0.02` did improve the same eval pack slightly:
| Baseline | Overall CER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: |
| raw tesseract (`psm=6`) | 0.329012 | 0.243379 | 0.757175 | 1.162917 |
| routed tesseract (`psm=6` vs `psm=13`, margin `0.02`) | 0.324000 | 0.242003 | 0.733983 | 1.116534 |
Paired CER on that same pack was `5` wins, `3` losses, and `172` ties against raw default Tesseract. This is worth keeping as a baseline candidate backend, but it is not strong enough to confuse with model leadership, and the routing margin was selected on the same held-out pack.
We then used the new calibration workflow to stop cheating on that threshold. The original `180`-sample mixed eval was split into a calibration half (`89` samples) and untouched holdout half (`91` samples). `psm=13` was benchmarked against raw default Tesseract on the calibration half, `calibrate-quality-router` selected `secondary_quality_margin=0.03`, and that routed candidate was evaluated on the untouched holdout:
| Baseline | Overall CER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: |
| raw tesseract (`psm=6`) | 0.342234 | 0.236933 | 0.835834 | 1.269163 |
| routed tesseract (`psm=6` vs `psm=13`, calibrated on separate half) | 0.340608 | 0.235094 | 0.835207 | 1.267909 |
Paired CER on the untouched holdout was `2` wins, `1` loss, and `88` ties. That is a real out-of-sample gain, but the size is too small to matter for the main project objective. It tells us the routing idea is not pure leakage, not that we have a strong OCR engine.
We also tested a richer calibrated ensemble over `psm=6/7/11/13`. The calibration half liked it and selected biases:
- `psm7=-0.12`
- `psm11=0.02`
- `psm13=-0.02`
But the untouched holdout rejected it:
| Baseline | Overall CER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: |
| raw tesseract (`psm=6`) | 0.342234 | 0.236933 | 0.835834 | 1.269163 |
| calibrated `psm=6/7/11/13` ensemble | 0.358352 | 0.246311 | 0.883548 | 1.375632 |
Surya by itself was not competitive on the same `91`-sample holdout. As a raw baseline it landed at overall CER `2.198362` versus raw Tesseract `0.342234`, mainly because it was disastrous on the English-heavy line-crop slice. But Surya was materially better on the Nepali slice, so we tested a calibrated script-aware route instead of discarding it outright.
That routed candidate uses raw Tesseract output as the primary signal: if the Tesseract text looks sufficiently Devanagari-heavy, switch to Surya; otherwise keep Tesseract. The threshold was calibrated on the separate `89`-sample half with `ocrtech calibrate-script-router`, which selected `primary_script_threshold=0.4`. On the untouched `91`-sample holdout:
| Baseline | Overall CER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: |
| raw tesseract (`psm=6`) | 0.342234 | 0.236933 | 0.835834 | 1.269163 |
| calibrated script router (`tesseract` -> `surya`) | 0.313084 | 0.230513 | 0.700133 | 1.172699 |
Paired CER on that holdout was `9` wins, `0` losses, and `82` ties against raw Tesseract. This is the strongest out-of-sample baseline improvement in the repo so far. It is still not claim-grade evidence, because it is one internal line-level split rather than the full internal document pack plus public suites, but it is a real improvement and a better direction than the TrOCR branch.
We also ran an exploratory `routing_granularity=page` variant of the same Tesseract-to-Surya router on that holdout, reusing the calibrated threshold `0.4`. It produced the same summary numbers as the document-level router. That is not a hidden win or loss; it just means this holdout is mostly single-page line-crop material, so page routing has no additional room to differentiate behavior yet.
After that, the internal eval pack was upgraded so image-mode tables are rendered as actual grids and the pack now includes a real `multi_page` slice as bundled page images. On that rendered `5`-sample smoke pack, the plain script router reached `table_cell_f1 = 1.0` on the dedicated table page but still missed the table inside the multi-page report when it kept Tesseract text for the document. A follow-up exploratory candidate with `secondary_structure_backfill=true` fixed that exact gap: it kept the same CER while backfilling Surya table structure on the multi-page report, moving aggregate `table_cell_f1` from `0.5` to `1.0` on the rendered smoke pack. This is still smoke evidence, not claim evidence, because the pack is tiny and only covers the `clean` slice, but it is the first concrete sign that composition can improve document structure without paying a text penalty.
The next run made that path less hand-wavy. A broader rendered image pack was generated with `25` samples across `clean`, `scan`, `blur`, `low_contrast`, and `skew`, and `audit-eval-pack --require-claim-ready` passed on it. The backfill router was then calibrated through `ocrtech calibrate-script-router` with `secondary_structure_backfill=true`, producing `ocrtech-tesseract-surya-script-router-backfill-calibrated`, and `preflight-claim` passed on the final experiment pack.
Pulled artifacts for that run:
- `outputs/live-k2-claimpack-rendered-backfill/eval-pack-audit.json`
- `outputs/live-k2-claimpack-rendered-backfill/eval-pack-audit.md`
The next useful tightening step was not “run everything on all `100` rendered samples and hope.” That is too expensive for the current GLM and PaddleOCR-VL paths on `k2`. Instead, the repo now has `ocrtech filter-manifest`, which was used to create a reproducible `5`-sample blur-only subset covering:
- paragraph
- table
- form
- figure
- multi-page report
That subset was benchmarked on `k2` across the real baseline set:
- `candidate` using `outputs/script-router-backfill-calibration/model-card.json`
- `tesseract`
- `surya`
- `stock-paddle`
- `glm-ocr` via the isolated `.venv-glm` hook
- `paddleocr-vl`
All `30` runs completed successfully. The pulled local artifacts are:
- `outputs/live-k2-rendered-blur5/report.json`
- `outputs/live-k2-rendered-blur5/summary.json`
Aggregate OCR/text result on that subset:
- candidate CER: `0.223273`
- Surya CER: `0.197601`
- Tesseract CER: `0.732364`
- stock-Paddle CER: `2.810426`
- GLM-OCR CER: `2.820269`
- PaddleOCR-VL CER: `18.470663`
Aggregate structure result on that subset:
- candidate `table_cell_f1`: `1.0`
- Surya `table_cell_f1`: `1.0`
- Tesseract / stock-Paddle / GLM-OCR / PaddleOCR-VL `table_cell_f1`: `0.0`
Aggregate reading-order result on that subset:
- candidate: `0.783636`
- Tesseract: `0.823636`
- Surya: `0.750909`
- stock-Paddle: `1.0`
- GLM-OCR: `1.0`
- PaddleOCR-VL: `0.750909`
That subset is exploratory, not claim-grade. But it is useful because it removes more guesswork:
- the current candidate is much better than Tesseract, stock-Paddle, GLM-OCR, and PaddleOCR-VL on blur-heavy rendered OCR CER
- the current candidate still does not beat raw Surya on text CER
- the current candidate still trades away some reading-order quality relative to Tesseract and the layout-heavy baselines
For paper use, the right interpretation is narrower than "our model wins." This subset supports a more defensible statement: the current backfill candidate is a strong internal engineering baseline on degraded rendered documents, especially for table recovery, but raw Surya is still the stronger text OCR baseline on the same subset.
Additional pulled artifacts for that `25`-sample backfill run:
- `outputs/live-k2-claimpack-rendered-backfill/calibration.json`
- `outputs/live-k2-claimpack-rendered-backfill/model-card.json`
- `outputs/live-k2-claimpack-rendered-backfill/experiment.json`
- `outputs/live-k2-claimpack-rendered-backfill/report.json`
- `outputs/live-k2-claimpack-rendered-backfill/summary.json`
- `outputs/live-k2-claimpack-rendered-backfill/summary.md`
Key numbers versus raw Tesseract on that rendered claim-ready pack:
| Slice | Candidate CER | Tesseract CER | Notes |
| --- | ---: | ---: | --- |
| all | 0.248654 | 0.689214 | paired CER `9` wins, `1` loss, `15` ties |
| clean | 0.243224 | 0.540584 | same win pattern as smoke pack |
| blur | 0.260039 | 0.991613 | strongest degraded-slice text gain |
| scan | 0.237424 | 0.593631 | clear text win |
| low_contrast | 0.263265 | 0.554174 | smaller but still positive |
| skew | 0.239317 | 0.766067 | one loss, two wins, two ties |
| multi_page | 0.315789 | 0.315789 | CER tie |
| table | 0.339713 | 1.421531 | large text gain on table-heavy pages |
Structure metrics on the same pack matter more than the CER win:
- candidate aggregate `table_cell_f1`: `0.955556`
- Tesseract aggregate `table_cell_f1`: `0.0`
- candidate aggregate `table_exact_match`: `0.8`
- Tesseract aggregate `table_exact_match`: `0.0`
- candidate `multi_page` `table_cell_f1`: `0.911111`
- Tesseract `multi_page` `table_cell_f1`: `0.0`
This is better evidence than the tiny smoke pack, but it still is not a clean claim story. Two real weaknesses remain on this rendered pack:
- `form` CER regressed slightly: `0.231933` vs `0.230010`
- aggregate `reading_order_pair_accuracy` is lower than Tesseract: `0.764` vs `0.816`
So the honest reading is narrow: the calibrated Tesseract-plus-Surya backfill router is a strong internal rendered-pack baseline for OCR robustness and table extraction, but it is not yet a broad document-understanding winner.
We then ran two more comparisons on that same rendered claim-ready pack to stop overfitting the story to Tesseract.
First, raw Surya on the rendered pack was not the disaster it had been on the older line-crop holdout. On the `25`-sample rendered document pack it beat the calibrated backfill router on text CER across every checked slice:
| Slice | Candidate CER | Raw Surya CER |
| --- | ---: | ---: |
| all | 0.248654 | 0.195260 |
| blur | 0.260039 | 0.202626 |
| scan | 0.237424 | 0.191862 |
| low_contrast | 0.263265 | 0.191862 |
| skew | 0.239317 | 0.194733 |
| form | 0.231933 | 0.138462 |
| multi_page | 0.315789 | 0.200000 |
| table | 0.339713 | 0.281818 |
Paired CER was not close: `0` wins, `15` losses, `10` ties for the candidate on `all`.
That does not mean the router is useless. Structure on the same pack is more mixed:
- aggregate `table_cell_f1`: candidate `0.955556`, Surya `0.955556`
- aggregate `reading_order_pair_accuracy`: candidate `0.764000`, Surya `0.750909`
- paired reading-order wins on `all`: candidate `2`, losses `0`, ties `23`
So the rendered document-pack conclusion is stricter than the earlier line-crop conclusion: the current routed candidate is still behind raw Surya on text OCR, while only slightly ahead on reading-order behavior and roughly tied on table structure.
Second, `stock-paddle` stopped being a fake baseline. The original `k2` benchmark result showing `1.0` CER everywhere was an adapter bug in this repo, not a real PaddleOCR result: `paddleocr 3.7.0` on `k2` returns v3 `OCRResult` objects, and the old parser only understood the older tuple/list shape. After fixing that adapter and installing `paddlepaddle 3.3.1`, the stock-Paddle baseline became runnable on `k2`.
With that fix in place, the same rendered claim-ready pack produced:
| Slice | Candidate CER | Stock-Paddle CER |
| --- | ---: | ---: |
| all | 0.248654 | 1.749081 |
| blur | 0.260039 | 1.549743 |
| scan | 0.237424 | 1.896558 |
| low_contrast | 0.263265 | 1.854006 |
| skew | 0.239317 | 1.568643 |
| form | 0.231933 | 0.470694 |
| multi_page | 0.315789 | 0.391388 |
| table | 0.339713 | 3.395694 |
Paired CER on `all` was `21` wins, `4` losses, `0` ties for the candidate.
Structure was lopsided in the other direction:
- candidate aggregate `table_cell_f1`: `0.955556`
- stock-Paddle aggregate `table_cell_f1`: `0.0`
- candidate aggregate `reading_order_pair_accuracy`: `0.764000`
- stock-Paddle aggregate `reading_order_pair_accuracy`: `1.0`
So stock-Paddle, as currently configured here with `PaddleOCR(lang='ne')`, preserves reading order very aggressively but is not competitive on text accuracy or table extraction for this mixed rendered pack.
The stricter rendered-document result is now also explicit. On `k2`, a larger `claimpack-rendered-v2` split was calibrated separately and evaluated on an untouched `50`-sample holdout. The routed script candidate selected `primary_script_threshold=0.0` on calibration, which means it collapsed to "always pick Surya" there. On the holdout:
| Baseline | Overall CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| candidate | 0.180893 | 0.947500 | 0.750909 |
| raw Surya | 0.180893 | 0.970833 | 0.750909 |
| raw Tesseract | 0.714207 | 0.000000 | 0.787818 |
Pulled local artifacts:
- `outputs/live-k2-claimpack-v2-script-router/calibration.json`
- `outputs/live-k2-claimpack-v2-script-router/model-card.json`
- `outputs/live-k2-claimpack-v2-script-router/experiment.json`
- `outputs/live-k2-claimpack-v2-script-router/summary.json`
- `outputs/live-k2-claimpack-v2-script-router/preflight.json`
- `outputs/paper-claimpack-v2-table/paper-table.md`
This is not a routed-model win. It is a branch ceiling. The candidate does not beat Surya on text, ties it exactly on CER, and is slightly worse on table structure. The new claim guardrail now rejects this case directly: `preflight-claim` fails `candidate-calibration` when a routed candidate collapses to one engine on every calibration sample.
We also tested the obvious opposite branch after adding Latin-script routing support to the repo: a Surya-primary page router with Tesseract fallback for English-heavy pages. The rendered-v2 calibration on `k2` selected `primary_script_threshold=0.7`, and the holdout result was unambiguously bad:
| Baseline | Overall CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| candidate | 0.416854 | 0.337500 | 0.761818 |
| raw Surya | 0.180893 | 0.970833 | 0.750909 |
| raw Tesseract | 0.714207 | 0.000000 | 0.787818 |
Pulled local artifacts:
- `outputs/live-k2-claimpack-v2-surya-latin-router/calibration.json`
- `outputs/live-k2-claimpack-v2-surya-latin-router/experiment.json`
- `outputs/live-k2-claimpack-v2-surya-latin-router/report.json`
- `outputs/live-k2-claimpack-v2-surya-latin-router/summary.json`
- `outputs/paper-claimpack-v2-surya-latin-table/paper-table.md`
Paired CER versus raw Surya on that holdout was `0` wins, `12` losses, `38` ties. So the inverted script router is not a refinement. It is a disproved branch.
We then tested the narrower quality-based version of the same idea: keep Tesseract and Surya as components, but let the router pick based on the local `quality_score` signal instead of script ratio. The rendered-v2 calibration selected `selected_threshold=0.0`, which already tells you the calibration split did not find a real decision boundary beyond "prefer Surya whenever it is not clearly worse." On the untouched holdout:
| Baseline | Overall CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| candidate | 0.181128 | 0.947500 | 0.750909 |
| raw Surya | 0.180893 | 0.947500 | 0.750909 |
| raw Tesseract | 0.714207 | 0.000000 | 0.787818 |
Pulled local artifacts:
- `outputs/live-k2-claimpack-v2-surya-quality-router/calibration.json`
- `outputs/live-k2-claimpack-v2-surya-quality-router/experiment.json`
- `outputs/live-k2-claimpack-v2-surya-quality-router/report.json`
- `outputs/live-k2-claimpack-v2-surya-quality-router/summary.json`
- `outputs/paper-claimpack-v2-surya-quality-router-table/paper-table.md`
Paired CER versus raw Surya on that holdout was `0` wins, `1` loss, `49` ties. So this is not the missing refinement either. It is nearly indistinguishable from Surya and still not better.
We also tested a non-routing composition on the same rendered-v2 holdout: `line_align_composite`. The idea was simple enough to be worth disproving. Keep Surya as the text engine, reuse Tesseract line scaffolding, and see whether better ordering helps enough to justify the merge. It did not:
| Baseline | Overall CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| candidate | 0.310549 | 0.897500 | 0.792727 |
| raw Surya | 0.180893 | 0.970833 | 0.750909 |
| raw Tesseract | 0.714207 | 0.000000 | 0.787818 |
Pulled local artifacts:
- `outputs/live-k2-claimpack-v2-line-align-summary/summary.json`
- `outputs/paper-claimpack-v2-line-align-table/paper-table.md`
- `outputs/live-k2-claimpack-v2-line-align-model-card.json`
Paired CER versus raw Surya on that holdout was `0` wins, `7` losses, `43` ties. The candidate did improve reading-order accuracy relative to raw Surya, and even edged past Tesseract there, but the text and table regressions are too large to defend. So line alignment is another useful negative result, not a candidate answer.
Finally, we measured the sample-level upper bound for the same component family using `benchmark-oracle`. This is not a deployable candidate. It chooses the better of raw Surya and raw Tesseract per sample using ground-truth CER. That is the right diagnostic question before wasting more time on routing heuristics:
| System | Overall CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| Surya/Tesseract oracle | 0.180529 | 0.970833 | 0.750909 |
| raw Surya | 0.180893 | 0.970833 | 0.750909 |
| raw Tesseract | 0.714207 | 0.000000 | 0.787818 |
Pulled/generated local artifacts:
- `outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-analysis.json`
- `outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-choices.json`
- `outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-summary/summary.json`
- `outputs/paper-claimpack-v2-oracle-table/paper-table.md`
The oracle picked raw Surya for `49` samples and raw Tesseract for `1`. Those per-sample source choices are stored in `oracle-choices.json` as provenance, not mixed into benchmark metrics. Paired CER versus raw Surya was `1` win, `0` losses, `49` ties. That means the current Surya/Tesseract routing space is effectively exhausted for text OCR on rendered-v2. The next useful model-side work needs a stronger recognizer or document model, not another selector over these two engines.
We then added `benchmark-error-analysis` to convert that failure into a training target instead of another routing guess. On the same rendered-v2 benchmark report, raw Surya's highest absolute CER slices are `figure` (`0.253157`) and `table` (`0.252781`). The hardest individual samples are dominated by table-ledger and figure-caption pages. The Surya-vs-oracle diagnostic confirms that Tesseract only improves one sample, `eval-form-notice-003-blur`, by `0.018182` CER. So the next serious model branch should target table/figure-heavy mixed-script recognition and document context, while treating Surya/Tesseract routing as exhausted.
We also exported those ranked failures back into a real manifest instead of a hand-maintained list. `data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl` is a `15`-sample `hard_eval` manifest built from the top Surya CER failures. It keeps the original reference metadata and adds `metadata.failure_analysis` with rank, metric, value, and comparison details. The local checkout does not include the rendered image assets, so the local audit reports missing image paths; the same manifest passes full file/reference audit on `k2` at `outputs/audit-claimpack-v2-surya-hard15/manifest-audit.json`, pulled locally to `outputs/audit-claimpack-v2-surya-hard15-k2/manifest-audit.json`.
The hard pocket now has an explicit validation config: `configs/validation-hard15-surya.json`. This uses conjunctive gate slices such as `["hard_eval", "table"]` and `["hard_eval", "figure"]`, so the table and figure gates cannot be satisfied by easier non-hard samples. As a negative control, the current line-align candidate fails this config on `outputs/live-k2-claimpack-v2-line-align/report.json`: hard-eval CER is `0.482039` versus Surya `0.348706`, hard-table CER is `0.613636` versus Surya `0.363636`, and the figure/table structure gates do not improve over Surya.
For model iteration, we generated a separate table/figure development pack on `k2`:
- manifest: `data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl`
- audit: `outputs/audit-eval-pack-table-figure-dev-v1-k2/eval-pack-audit.json`
- samples: `80`
- slices: `40` table, `40` figure, `80` Nepali/English/mixed-script/image/reading_order
- degradations: `clean`, `scan`, `blur`, `low_contrast`, `skew`
- hard15 overlap: `0` sample IDs and `0` text hashes
This pack was generated with `--template table-ledger --template figure-caption --variant-offset 100`, after updating the eval-pack generator so table OCR references include visible table rows and figure variants include copy-specific visible text. Treat this as a development/training target, not as held-out claim evidence.
The pack was then split with text-level grouping so degraded variants of the same target do not leak across train/eval:
```bash
ocrtech split-manifest data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \
--out data/splits-table-figure-dev-v1 \
--eval-ratio 0.2 \
--seed 17 \
--stratify-by document_type \
--group-by text_sha256
```
Verified split:
- train: `60` samples
- eval: `20` samples
- text groups: `16`
- leakage check: `pass`
- train audit on `k2`: `pass`
- eval audit on `k2`: `pass`
Additional leakage checks on `k2` found no sample-id or text-hash overlap with `hard15` or rendered-v2 eval. That makes this a legitimate development pack for the next model branch, but it is still not claim evidence.
The first model-side attempt on this branch was a minimal Hugging Face VisionEncoderDecoder fine-tune using `microsoft/trocr-small-printed`:
```bash
ocrtech train-recognizer \
--backend hf-vision-encoder-decoder \
--base-model microsoft/trocr-small-printed \
--train-manifest data/splits-table-figure-dev-v1/train.jsonl \
--eval-manifest data/splits-table-figure-dev-v1/eval.jsonl \
--out runs/hf-table-figure-dev-v1 \
--epochs 1 \
--train-batch-size 4 \
--eval-batch-size 4 \
--learning-rate 5e-5 \
--run
```
Training completed and the package audited cleanly, but the result was a decisive negative:
| metric | value |
| --- | ---: |
| eval_loss | 7.884517 |
| eval_cer | 3.062119 |
| eval_wer | 3.562689 |
| eval_exact_line_accuracy | 0.000000 |
Benchmarking the packaged candidate against Surya and Tesseract on the `20`-sample dev eval confirmed the branch is not viable:
| system | CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| HF table/figure candidate | 2.422001 | 0.000000 | 0.000000 |
| Surya | 0.563399 | 0.858333 | 0.550000 |
| Tesseract | 0.546227 | 0.000000 | 0.695000 |
Paired CER was `0` wins, `20` losses, and `0` ties against both Surya and Tesseract. The failure mode is not subtle: naive full-page TrOCR fine-tuning does not learn table/figure document recognition from this small pack. The next credible recognizer branch should train on line/cell crops or use a page/document model with structure-aware supervision.
We then implemented that crop-level path and tested it instead of leaving it as advice. Newly generated image-pack references now carry renderer layout metadata for non-skew pages, and `ocrtech extract-crops` turns those verified bboxes into ordinary recognizer manifests. Skew pages are skipped because pre-rotation bboxes are invalid.
Full crop development pack on `k2`:
- source pack: `data/eval-pack-table-figure-crops-dev-v1/manifests/ocrtech-eval-pack.jsonl`
- crop manifest: `data/crops-table-figure-dev-v1/manifests/crops.jsonl`
- source samples: `80`
- crops: `544`
- crop types: `384` table cells, `128` lines, `32` figure captions
- skipped candidates: `32` skew pages
- crop audit: `outputs/audit-crops-table-figure-dev-v1/manifest-audit.md`
- split: `480` train / `64` eval
- split grouping: `text_sha256`
- split leakage: pass
The first crop-level HF model used the same `microsoft/trocr-small-printed` backend for one epoch, this time on `480` crop images instead of full pages. Trainer metrics improved relative to the full-page branch, but still were not competitive:
| metric | value |
| --- | ---: |
| eval_loss | 3.087561 |
| eval_cer | 0.959831 |
| eval_wer | 1.363715 |
| eval_exact_line_accuracy | 0.078125 |
Packaged crop model audit passed:
- `runs/hf-crops-table-figure-dev-v1/training_metrics.json`
- `runs/hf-crops-table-figure-dev-v1/package/model-card.json`
- `outputs/audit-hf-crops-table-figure-dev-v1-model/model-audit.md`
Benchmark result on the `64`-sample crop eval:
| system | CER |
| --- | ---: |
| HF crop candidate | 1.043369 |
| Surya | 0.475309 |
| Tesseract | 0.077453 |
Paired CER:
- versus Surya: `7` wins, `50` losses, `7` ties
- versus Tesseract: `0` wins, `64` losses, `0` ties
This means crop-level training is the right data shape, but this tiny HF recognizer is still the wrong recognizer. The immediate technical takeaway is that Tesseract is extremely strong on these generated crop images; the next branch should use strong crop OCR behind better layout/cell/caption detection, or train a more suitable recognizer with substantially more real crop data.
The next composition test used that crop-OCR signal more directly: `table_cell_refine_composite` keeps Surya as the document/table structure engine and rereads detected table cells with Tesseract crops.
Two hard15 variants were tested:
| variant | hard_eval CER | table_cell_f1 | reading_order_pair_accuracy | verdict |
| --- | ---: | ---: | ---: | --- |
| Surya baseline | 0.348706 | 0.979167 | 0.520000 | current hard15 target |
| `replacement_policy=always` | 0.348706 | 0.593750 | 0.520000 | unsafe; damages good Surya cells |
| `replacement_policy=empty` | 0.348706 | 0.979167 | 0.520000 | safe tie; no improvement |
| `replacement_policy=empty`, table de-dup | 0.348706 | 0.979167 | 0.520000 | parser hygiene fix; still no improvement |
The safe policy also failed the hard15 validation gate with status `not_validated`. This is another useful negative result: Tesseract crop OCR is strong when the crop is exact, but naively applying it to Surya-inferred table cells does not improve Surya on this hard pocket.
Cell-level analysis explains why:
| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| Surya | 96 | 94 | 0.979167 | 0.010417 |
| table-cell refine, `always` | 96 | 57 | 0.593750 | 0.284309 |
| table-cell refine, `empty` | 96 | 94 | 0.979167 | 0.010417 |
Against Surya, the unsafe `always` policy improved `0` cells, worsened `37`, and tied `59`. The safe `empty` policy tied all `96` cells because Surya had no missing cells to fill in this hard pocket. The remaining Surya cell errors are only two occurrences of `किरण` predicted as `करिण`, both on the same table-ledger variant. The next table branch needs a targeted confidence/replacement signal for rare wrong-but-nonempty cells; blind crop rerecognition is the wrong direction.
The table de-duplication branch fixed a separate parser hygiene issue: line-based table inference no longer creates a second table when an engine already supplied an overlapping structured table. On the rerun artifact, both candidate and Surya now report `8` predicted tables for `8` reference tables and `0` duplicate overlapping tables. This is useful for cleaner exports, but because the fix applies at parser assembly time it also benefits the Surya baseline; hard15 validation remains `not_validated`.
Generated diagnostics:
- `outputs/paper-claimpack-v2-surya-error-analysis/error-analysis.md`
- `outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json`
- `outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis/error-analysis.md`
- `outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis/error-analysis.md`
- `data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl`
- `configs/validation-hard15-surya.json`
- `outputs/validation-claimpack-v2-line-align-hard15/validation.json`
- `outputs/audit-claimpack-v2-surya-hard15-k2/manifest-audit.json`
- `data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-eval-pack-table-figure-dev-v1-k2/eval-pack-audit.json`
- `data/splits-table-figure-dev-v1/split-summary.md`
- `outputs/audit-table-figure-dev-v1-train-k2/manifest-audit.md`
- `outputs/audit-table-figure-dev-v1-eval-k2/manifest-audit.md`
- `runs/hf-table-figure-dev-v1/training_metrics.json`
- `runs/hf-table-figure-dev-v1/package/model-card.json`
- `outputs/audit-hf-table-figure-dev-v1-model/model-audit.md`
- `outputs/summary-hf-table-figure-dev-v1/summary.md`
- `outputs/experiment-hf-table-figure-dev-v1/benchmark/report.json`
- `outputs/crops-table-figure-dev-v1-k2/crop-manifest-summary.md`
- `outputs/crops-table-figure-dev-v1-k2/manifest-audit.md`
- `data/splits-crops-table-figure-dev-v1/split-summary.md`
- `runs/hf-crops-table-figure-dev-v1/training_metrics.json`
- `runs/hf-crops-table-figure-dev-v1/package/model-card.json`
- `outputs/audit-hf-crops-table-figure-dev-v1-model/model-audit.md`
- `outputs/summary-hf-crops-table-figure-dev-v1/summary.md`
- `outputs/experiment-hf-crops-table-figure-dev-v1/benchmark/report.json`
- `outputs/table-cell-refine-v1/model-card.json`
- `outputs/summary-table-cell-refine-hard15-v1/summary.md`
- `outputs/table-cell-refine-empty-v1/model-card.json`
- `outputs/summary-table-cell-refine-empty-hard15-v1/summary.md`
- `outputs/validation-table-cell-refine-empty-hard15-v1/validation.md`
- `outputs/table-cell-analysis-refine-always-hard15-v1/table-cell-analysis.md`
- `outputs/table-cell-analysis-refine-empty-hard15-v1/table-cell-analysis.md`
- `outputs/table-cell-refine-empty-dedup-v1/model-card.json`
- `outputs/summary-table-cell-refine-empty-dedup-hard15-v3/summary.md`
- `outputs/table-cell-analysis-refine-empty-dedup-hard15-v3/table-cell-analysis.md`
- `outputs/validation-table-cell-refine-empty-dedup-hard15-v3/validation.md`
When repeating this workflow, keep the comparison honest:
```bash
ocrtech prepare-data gauravgiri/nepali-ocr-dataset --hf --split train --limit 100 --skip-invalid --out data/manifests
ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines --hf --split train --limit 100 --out data/manifests
ocrtech prepare-data nvidia/OCR-Synthetic-Multilingual-v1 --hf --hf-subset en --split test --limit 1000 --out data/manifests-english-test
ocrtech merge-manifests \
data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl \
data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl \
data/manifests-english-test/nvidia__OCR-Synthetic-Multilingual-v1__en-test.jsonl \
--out data/manifests/nep-eng-mixed-v1.jsonl
ocrtech split-manifest data/manifests/nep-eng-mixed-v1.jsonl --out data/splits-mixed-v1 --eval-ratio 0.15 --seed 13
ocrtech materialize-references data/splits-mixed-v1/eval.jsonl --out data/references-mixed-v1 --rewritten-manifest data/splits-mixed-v1/eval-with-refs.jsonl
ocrtech rebalance-manifest data/splits-mixed-v1/train.jsonl --out data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl --slice english --slice nepali --seed 13
ocrtech train-recognizer --backend hf-vision-encoder-decoder --base-model microsoft/trocr-small-printed --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl --out runs/hf-recognizer-mixed-v1-rebalanced --train-batch-size 4 --eval-batch-size 4 --epochs 0.5 --run
ocrtech package-hf-recognizer --model-dir runs/hf-recognizer-mixed-v1-rebalanced/final-model --out runs/hf-recognizer-mixed-v1-rebalanced/package --model-id ocrtech-hf-mixed-v1-rebalanced --base-model microsoft/trocr-small-printed --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl --training-summary runs/hf-recognizer-mixed-v1-rebalanced/recognizer-training-summary.json --metrics-report runs/hf-recognizer-mixed-v1-rebalanced/training_metrics.json
ocrtech run-experiment --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl --out outputs/experiment-hf-mixed-v1-rebalanced --baselines candidate --candidate-model-config runs/hf-recognizer-mixed-v1-rebalanced/package/model-card.json --references-dir data/references-mixed-v1 --skip-validation
```
## Figure Caption Refinement Composite
The figure-caption hard15 slice exposed a parser problem first: after `[Figure: ...]`, the actual caption line was being rejected when OCR damaged the Nepali caption keyword (`चित्र` became variants such as `चतुर्थ` or `चत्रि`). The parser now accepts a nearby line after a figure marker as the caption if it is spatially close and not another marker/table line. That reduced Surya/candidate figure-caption CER on hard15 from about `0.737609` to `0.092323`.
The next branch used that cleaner structure and tried a narrow composite: keep Surya as the primary document engine, but replace only overlapping figure-caption lines with Tesseract when the Tesseract line has a better caption-prefix score. The replacement keeps Surya's layout bbox so skewed Tesseract boxes do not break caption attachment.
Hard15 v2 result:
| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
| figure-caption refine | 0.228008 | 0.052478 | 1.000000 | 0.979167 | 0.520000 |
Validation against the hard15 Surya config is still `not_validated`, but one gate now passes:
- `hard_eval,figure` `figure_caption_cer`: pass, `6` wins / `0` losses / `1` tie, candidate `0.052478`, Surya `0.092323`, improvement `0.039845`
- overall hard_eval CER: fail, improvement `0.008830 < 0.020000`
- hard_eval figure CER: fail, improvement `0.018922 < 0.020000`
- table CER/F1 gates: fail, ties only
Artifacts:
- `outputs/figure-caption-refine-v1/model-card.json`
- `outputs/audit-figure-caption-refine-v1-model/model-audit.md`
- `outputs/summary-figure-caption-refine-hard15-v2/summary.md`
- `outputs/experiment-figure-caption-refine-hard15-v2/benchmark/report.json`
- `outputs/validation-figure-caption-refine-hard15-v2/validation.md`
This is the first branch in this sequence that beats Surya on a configured gate, but it is a slice win, not a SOTA claim.
The stronger opt-in structural-label branch adds edit-distance guarded correction for document-role labels (`वार्षिक प्रतिवेदन`, `चित्र`, `निष्कर्ष`) after caption replacement:
| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| figure-caption refine + labels | 0.204610 | 0.008746 | 1.000000 | 0.979167 | 0.520000 |
Validation still reports `not_validated`, but the pass/fail split is materially better:
- overall hard_eval CER: pass, `7` wins / `0` losses / `8` ties, candidate `0.204610`, Surya `0.236839`, improvement `0.032228`
- `hard_eval,figure` CER: pass, `7` wins / `0` losses / `0` ties, candidate `0.022866`, Surya `0.091927`, improvement `0.069061`
- `hard_eval,figure` `figure_caption_cer`: pass, `7` wins / `0` losses / `0` ties, candidate `0.008746`, Surya `0.092323`, improvement `0.083576`
- table CER/F1 gates: fail, ties only
Artifacts:
- `outputs/figure-caption-refine-labels-v1/model-card.json`
- `outputs/audit-figure-caption-refine-labels-v1-model/model-audit.md`
- `outputs/summary-figure-caption-refine-labels-hard15-v1/summary.md`
- `outputs/experiment-figure-caption-refine-labels-hard15-v1/benchmark/report.json`
- `outputs/validation-figure-caption-refine-labels-hard15-v1/validation.md`
The next hard15 candidate added two explicitly auditable pieces on top of that branch:
- a small curated table-name lexicon, `configs/nepali-table-name-lexicon-v1.txt`, used only for cells under a header close to `नाम`
- a guarded structural-title correction for the recurring table title `दैनिक हिसाब`
This v3 run is the strongest internal hard15 result so far:
| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
| figure-caption + labels + table lexicon | 0.010671 | 0.008746 | 1.000000 | 1.000000 | 0.520000 |
Validation remains `not_validated`, but only one configured gate now fails:
- overall hard_eval CER: pass, `15` wins / `0` losses / `0` ties, candidate `0.010671`, Surya `0.236839`
- `hard_eval,table` CER: pass, `8` wins / `0` losses / `0` ties, candidate `0.000000`, Surya `0.363636`
- `hard_eval,figure` CER: pass, `7` wins / `0` losses / `0` ties, candidate `0.022866`, Surya `0.091927`
- `hard_eval,figure` `figure_caption_cer`: pass, `7` wins / `0` losses / `0` ties, candidate `0.008746`, Surya `0.092323`
- `hard_eval,table` `table_cell_f1`: fail, candidate `1.000000`, Surya `0.979167`, improvement `0.020833`, but bootstrap CI low is `0.000000` because the 8-page table slice has only 2 improving table pages and 6 ties
Cell-level analysis is still useful evidence:
- candidate: `96/96` exact cells, mean cell CER `0.000000`
- Surya: `94/96` exact cells, mean cell CER `0.010417`
- Tesseract full-page parser: `0/96` structured cells
- paired candidate-vs-Surya cells: `2` candidate-better, `0` Surya-better, `94` ties
Artifacts:
- `outputs/figure-caption-table-lexicon-v1/model-card.json`
- `outputs/audit-figure-caption-table-lexicon-v1-model/model-audit.md`
- `outputs/experiment-figure-caption-table-lexicon-hard15-v3/benchmark/report.json`
- `outputs/summary-figure-caption-table-lexicon-hard15-v3/summary.md`
- `outputs/table-cell-analysis-figure-caption-table-lexicon-hard15-v3/table-cell-analysis.md`
- `outputs/validation-figure-caption-table-lexicon-hard15-v3/validation.md`
Important interpretation: the hard15 table-page reference `text` is the page title (`दैनिक हिसाब`), while table contents are evaluated through structured table references. A temporary v2 attempt to include table grids in `Document.text` was rejected after verifying the actual reference schema. We did not weaken validation gates after observing the result.
## Table-Ledger Claim Pack V2
To test whether the table-cell gain survives a larger table-only pack, we generated an independent rendered image pack:
```bash
ocrtech create-eval-pack \
--out data/eval-pack-table-ledger-claim-v1 \
--input-format image \
--template table-ledger \
--count-per-template 12 \
--variant-offset 300 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast \
--degradation skew
```
This produced `60` table-ledger image samples. The pack audited cleanly but is intentionally not broad claim-ready because it covers one document type.
Validation config: `configs/validation-table-ledger-surya.json`.
Validation result: `validated` after adding a grid-structure non-regression gate.
| gate | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 60 | 15 | 0 | 45 | 1.000000 | 0.979167 | 0.020833 | 0.012500 |
| table_exact_match | 60 | 15 | 0 | 45 | 1.000000 | 0.750000 | 0.250000 | 0.150000 |
| table_structure_similarity | 60 | 0 | 0 | 60 | 1.000000 | 1.000000 | 0.000000 | 0.000000 |
Cell-level analysis:
| system | reference_tables | reference_cells | exact_cells | exact_rate | missing_cells | mean_cer |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| candidate | 60 | 720 | 720 | 1.000000 | 0 | 0.000000 |
| Surya | 60 | 720 | 705 | 0.979167 | 0 | 0.005208 |
| Tesseract | 60 | 720 | 0 | 0.000000 | 715 | 0.999868 |
Runtime on k2:
| system | mean latency seconds | median latency seconds | max latency seconds |
| --- | ---: | ---: | ---: |
| candidate | 3.017929 | 2.967943 | 5.104267 |
| Surya | 2.478481 | 2.453125 | 3.175586 |
| Tesseract | 0.146866 | 0.134735 | 0.301695 |
Artifacts:
- `data/eval-pack-table-ledger-claim-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-table-ledger-claim-v1/eval-pack-audit.md`
- `outputs/experiment-table-ledger-claim-v2/benchmark/report.json`
- `outputs/summary-table-ledger-claim-v2/summary.md`
- `outputs/table-cell-analysis-table-ledger-claim-v2/table-cell-analysis.md`
- `outputs/validation-table-ledger-claim-v2b/validation.md`
Interpretation:
- This is now statistically validated evidence that the lexicon-assisted candidate improves table cell-content extraction over Surya on generated table-ledger pages while preserving Surya's already-perfect simple grid structure.
- It is not a broad OCR claim. Page CER on this targeted pack is diagnostic only because the parser does not currently render structured table blocks into `Document.text`, while the generated references include visible table rows in `text`.
- The candidate remains lexicon-assisted; the paper must state that the table-name correction uses a small curated lexicon and suffix-preserving matching for generated IDs such as `किरण-302`.
- `table_structure_similarity` can score rowspans/colspans when references use structured cells, but this table-ledger pack is simple-grid only and the metric is still not true TEDS over merged-cell HTML trees.
The eval-pack generator now includes a `complex-table-ledger` template that writes span-aware references with a merged header (`colspan=2`). This makes the validation path capable of catching a model that gets flattened cell text right but loses table span structure. We have not yet run a k2 claim experiment on this template; that should be the next table-specific benchmark before discussing complex-table behavior in the paper.
## Complex-Table Pack V1
We generated and benchmarked a targeted complex-table pack:
```bash
ocrtech create-eval-pack \
--out data/eval-pack-complex-table-v1 \
--input-format image \
--template complex-table-ledger \
--count-per-template 10 \
--variant-offset 500 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast
```
The pack has `40` samples and span-aware references with a merged header. It audited cleanly but is targeted development evidence, not broad claim-ready coverage.
Validation config: `configs/validation-complex-table-surya.json`.
Validation result: `validated` under a deliberately modest contract: improve cell-content F1 and do not regress exact/span structure against Surya.
| gate | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 8 | 0 | 32 | 0.928333 | 0.915000 | 0.013333 | 0.005000 |
| table_exact_match | 40 | 0 | 0 | 40 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| table_structure_similarity | 40 | 0 | 0 | 40 | 0.875000 | 0.875000 | 0.000000 | 0.000000 |
Cell analysis:
| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 557 | 0.928333 | 0.067667 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |
Artifacts:
- `data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-complex-table-v1/eval-pack-audit.md`
- `outputs/experiment-complex-table-v1/benchmark/report.json`
- `outputs/summary-complex-table-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-v1/table-cell-analysis.md`
- `outputs/validation-complex-table-v1/validation.md`
Interpretation:
- The candidate improves some table cell text over Surya because the lexicon correction now works below grouped headers.
- Neither candidate nor Surya reconstructs the merged-header span; both have `table_exact_match = 0` and `table_structure_similarity = 0.875`.
- This is not a complex-table SOTA result. It is a concrete failure analysis result showing the next engineering target: preserve or infer rowspans/colspans in parser output.
## Complex-Table Span-Inference V1
After adding repeated-header colspan inference to the candidate parser, we reran the same complex-table pack:
```bash
ocrtech run-experiment \
--eval-manifest data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl \
--out outputs/experiment-complex-table-span-infer-v1 \
--baselines candidate,surya,tesseract \
--candidate-model-config outputs/figure-caption-table-lexicon-v1/model-card.json \
--validation-config configs/validation-complex-table-surya.json
```
Then:
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/experiment-complex-table-span-infer-v1/benchmark/report.json \
--out outputs/summary-complex-table-span-infer-v1 \
--baseline surya \
--baseline tesseract \
--metric table_cell_f1 \
--metric table_exact_match \
--metric table_structure_similarity
ocrtech analyze-table-cells \
--benchmark-report outputs/experiment-complex-table-span-infer-v1/benchmark/report.json \
--eval-manifest data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl \
--out outputs/table-cell-analysis-complex-table-span-infer-v1 \
--baseline surya \
--baseline tesseract
```
Validation result: `validated`.
| gate | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |
Cell analysis:
| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 597 | 0.995000 | 0.001000 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |
Artifacts:
- `outputs/experiment-complex-table-span-infer-v1/benchmark/report.json`
- `outputs/experiment-complex-table-span-infer-v1/validation/validation.md`
- `outputs/summary-complex-table-span-infer-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-span-infer-v1/table-cell-analysis.md`
Interpretation:
- The candidate now reconstructs the generated merged header pattern via a narrow repeated-header span inference rule.
- This is a parser/postprocessing improvement, not a trained model improvement.
- It does not solve broad table understanding. We still need real scanned table benchmarks, varied spans/rowspans, and true HTML/TEDS scoring.
## Complex-Table TEDS-Style Metrics V1
We added two deterministic ordered table-tree edit metrics:
- `table_teds_structure`: compares `table/tr/td` tree shape plus `rowspan`/`colspan`
- `table_teds_text`: also includes normalized cell text in `td` node labels
Then we reran the same complex-table pack with stricter validation gates:
```bash
ocrtech run-experiment \
--eval-manifest data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl \
--out outputs/experiment-complex-table-teds-v1 \
--baselines candidate,surya,tesseract \
--candidate-model-config outputs/figure-caption-table-lexicon-v1/model-card.json \
--validation-config configs/validation-complex-table-surya.json
```
Validation result: `validated`.
| gate | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |
| table_teds_structure | 40 | 40 | 0 | 0 | 1.000000 | 0.904762 | 0.095238 | 0.095238 |
| table_teds_text | 40 | 40 | 0 | 0 | 0.996250 | 0.891667 | 0.104583 | 0.098690 |
Cell analysis is unchanged from the span-inference run:
| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 597 | 0.995000 | 0.001000 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |
Artifacts:
- `outputs/experiment-complex-table-teds-v1/benchmark/report.json`
- `outputs/experiment-complex-table-teds-v1/validation/validation.md`
- `outputs/summary-complex-table-teds-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-teds-v1/table-cell-analysis.md`
Interpretation:
- This is stronger than the old span-token score because it evaluates ordered table tree structure.
- It is still an internal deterministic metric over a generated pack. A broad paper claim still needs public benchmark scoring and real scanned tables with diverse merged-cell patterns.
## Quality Report
Every `parse-doc` output now includes:
```text
quality.json
```
The quality report tracks:
- line/block/table counts
- average and minimum OCR confidence
- missing confidence rates
- low confidence rates
- malformed table/block warnings
- `quality_score`
- fallback trigger/success/failure diagnostics when `metadata.fallback` exists
`quality_score` is useful as a diagnostic gate. It is not a substitute for CER/WER, table metrics, reading order metrics, or public benchmark results.
Figure/caption extraction is scored separately with `figure_detection_f1` and `figure_caption_cer` when structured references include `figures`.
Figure artifacts are exported explicitly. If an engine returns `figure.image_path`,
the parser copies that image into `figures/`; if only a figure bbox is available
and the source is a regular image or page-bundle directory, it crops
`figures/{figure_id}.png` when Pillow is installed. For bundles, it uses
`figure.page_index` to select the page image. When an artifact cannot be
produced, the figure metadata gets `metadata.export_warnings`. This makes figure
detections auditable as visual artifacts instead of only JSON boxes.
Selective fallback usage is exposed in benchmark reports as diagnostic metrics:
- `fallback_triggered`
- `fallback_succeeded`
- `fallback_failed`
- `fallback_primary_average_confidence`
- `fallback_threshold`
- `fallback_primary_quality_score`
- `fallback_min_quality_score`
These are not quality scores by themselves. They are audit fields that make it
possible to report how much of a claimed system came from the local engine and
how much came from a fallback such as PaddleOCR-VL.
Claim review can also cap `fallback_triggered` by slice with
`max_candidate_fallback_trigger_rate_by_slice`. Use this for SOTA runs to keep
the VLM fallback selective: high fallback use on figure or hard document slices
is a different claim from high fallback use on ordinary English/Nepali text
pages.
Benchmark rows now also include an `output_artifacts` object for successful
runs. It records the path, SHA-256, and byte size for exported files such as
`document.json`, `document.md`, `quality.json`, `tables/*.csv`,
`tables/*.html`, `figures/metadata.json`, and figure image crops. This does not
change any metric, but it makes every scored row point at the concrete
structured artifacts behind the score.
Benchmark rows also include `resource_metrics` when the host exposes process
resource usage. The current fields are `process_max_rss_mb` and
`process_max_rss_delta_mb`; `summarize-benchmark` aggregates them so paper
tables can report memory alongside `latency_seconds`. This is process RSS
telemetry, not GPU memory telemetry.
For CUDA or VLM benchmark hosts, pass `--capture-gpu-metrics` to sample
`nvidia-smi` before and after each row. When available, rows also record
`gpu_memory_used_mb`, `gpu_memory_total_mb`, `gpu_memory_used_delta_mb`, and
`gpu_utilization_percent`.
# Experiment Workflow
If you are writing a paper from these runs, read [PAPER_NOTES.md](PAPER_NOTES.md) alongside this file. This document is the raw chronology; `PAPER_NOTES.md` is the stricter claim ledger.
`run-experiment` is the reproducible wrapper around benchmark and validation. It records the manifest hash, train manifest hashes, environment metadata, baseline list, benchmark report path, and validation status.
## Smoke Experiment
```bash
ocrtech create-eval-pack --out data/eval-pack-smoke --input-format text
ocrtech run-experiment \
--eval-manifest data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack-smoke/references \
--baselines ours \
--skip-validation \
--out outputs/smoke-experiment
```
This proves the pipeline and scoring path works. It is not OCR evidence.
## Claim Experiment
Run preflight first:
```bash
ocrtech preflight-claim \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--candidate-model-config runs/recognizer/package/model-card.json \
--validation-config validation.json \
--train-manifest data/splits/train.jsonl \
--out outputs/preflight
```
If the candidate is a routed composite, do not under-specify the baseline list. A claim-grade `script_select_composite` run must include its raw components in the same experiment, for example `candidate,tesseract,surya`, otherwise `preflight-claim` now fails `candidate-components`.
```bash
ocrtech run-experiment \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--candidate-model-config runs/recognizer/package/model-card.json \
--validation-config validation.json \
--train-manifest data/splits/train.jsonl \
--preflight \
--require-trained-recognizer \
--require-model-admission \
--out outputs/claim-experiment \
--require-validated
```
`--preflight` runs the same claim-readiness checks inside `run-experiment` and
stops before benchmark output is produced if the setup cannot satisfy the
configured gates. `--require-validated` returns a non-zero exit code unless
validation status is `validated`.
After the internal claim experiment and the public suites are all complete, run `review-claim`; see [CLAIM_REVIEW.md](CLAIM_REVIEW.md).
Use `summarize-benchmark` when you need aggregate and paired slice comparisons from a raw `benchmark/report.json`:
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/claim-experiment/benchmark/report.json \
--out outputs/claim-experiment/benchmark-summary \
--candidate candidate \
--baseline tesseract \
--metric cer
```
Use `paper-benchmark-table` when you need one paper-facing table merged from multiple `summary.json` artifacts, for example separate candidate-vs-Tesseract, candidate-vs-Surya, and candidate-vs-stock-Paddle summaries from the same pack:
```bash
ocrtech paper-benchmark-table \
--summary outputs/paper-claimpack-summary-tesseract/summary.json \
--summary outputs/paper-claimpack-summary-surya/summary.json \
--summary outputs/paper-claimpack-summary-stock-paddle/summary.json \
--out outputs/paper-claimpack-table \
--system candidate \
--system surya \
--system tesseract \
--system stock-paddle \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--label 'stock-paddle=Stock Paddle' \
--source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' \
--source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' \
--source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' \
--source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds
```
The `--source-preference` pins each system to one summary artifact. That matters when the same candidate was rerun in separate comparisons and latency differs across those reruns.
## Outputs