prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
31.8 kB
# Validation Design
The claim target is not "we ran OCR and it looked good." The claim target is:
```text
ocrtech is better than named baselines on held-out Nepali + English OCR and document-structure extraction, under measured gates.
```
Until the validation report says `validated`, do not claim SOTA.
Claim-grade `validation.json` files record provenance for the benchmark report,
validation config, eval manifest, train manifests, candidate model card, and
fallback model card used by `validate-claim`. `review-claim` cross-checks
those hashes against `experiment.json`, so a validation artifact from a
different benchmark, eval split, validation config, model card, fallback card,
or training-manifest set cannot be reused as portfolio evidence.
The SOTA review profile is paired with these claim-grade validation configs:
- `configs/validation-internal-sota-v1.json`
- `configs/validation-public-omnidocbench-sota-v1.json`
- `configs/validation-public-real5-sota-v1.json`
Each one covers the `required_validation_gates` declared in
`configs/claim-review-sota-v1.json`. Use the slot-specific config when running
`run-experiment` or `validate-claim`; otherwise portfolio review may have
metrics and baselines but still lack the passed gates required for the paper
claim.
The built-in fallback validation config is intentionally broader than a plain
OCR smoke check. If `validate-claim` is called without `--config`, it now gates
Nepali CER/WER, English non-regression, table cell and TEDS-text quality,
reading order, document reconstruction, and figure detection/caption quality.
That default remains smaller than the SOTA configs above, but it should not
allow a "document OCR" claim to pass on text-only evidence.
## Evidence Required
A defensible run needs all of this:
- A claim-ready image eval pack from `ocrtech create-eval-pack`, plus public benchmark runs.
- A candidate model card from `ocrtech create-model-card` and a passing `ocrtech audit-model` report.
- Training data prepared with `audit-manifest` and `split-manifest`.
- Held-out eval manifest that was never used for recognizer, corrector, threshold, or prompt tuning.
- Image/PDF eval inputs normalized to OCR-supported file suffixes before any
claim-grade PaddleOCR/Tesseract/Surya/VLM comparison.
- For routed or thresholded candidates, a separate calibration experiment artifact that records how the threshold was chosen.
- Train manifests for leakage checks.
- Paired benchmark outputs for every baseline on the same input files.
- Per-sample metrics, not only aggregate averages.
- For `glm-ocr` and `paddleocr-vl`, a real runnable command hook that emits Markdown or `document.md`, not a placeholder env var.
- Slice coverage for Nepali, English, multi-page documents, tables, reading order, figures/charts, forms, scans, photos, skew, blur, compression, and low contrast.
- Gates that name the baseline, metric, slice, direction, required margin, and confidence requirement.
- Paired win/loss evidence, a minimum paired win-rate requirement when the
claim needs broad per-document improvement, and a nonparametric significance
check when the claim requires it.
- Failure-case report, not just the winning average.
## Public Benchmark Gate
Generated eval packs are development evidence. They are not enough for SOTA
wording. Before a paper claim can say "state of the art," at least one public
document parsing benchmark must be converted into an auditable manifest and run
as a normal `ocrtech run-experiment`.
Known public benchmark registry:
```bash
ocrtech list-public-benchmarks
```
Current registry entries:
- `opendatalab/OmniDocBench`: real-world document parsing benchmark, dataset page `https://huggingface.co/datasets/opendatalab/OmniDocBench`
- `allenai/olmOCR-bench`: PDF-to-Markdown benchmark with unit-test style assertions, dataset page `https://huggingface.co/datasets/allenai/olmOCR-bench`
- `PaddlePaddle/Real5-OmniDocBench`: real-world physical reconstruction benchmark with scanning, warping, screen-photo, illumination, and skew scenarios, dataset page `https://huggingface.co/datasets/PaddlePaddle/Real5-OmniDocBench`
The first implemented public converter is:
```bash
ocrtech prepare-public-benchmark opendatalab/OmniDocBench \
--out data/public/omnidocbench \
--limit 100 \
--language english \
--require-tables
```
The converter was written only after checking the actual dataset structure:
`OmniDocBench.json` contains `page_info`, `page_info.page_attribute`, and
`layout_dets`; image files resolve as `images/{page_info.image_path}`; table
ground truth is stored in table-block `html`.
Every public benchmark manifest row must include:
- `metadata.public_benchmark = true`
- `metadata.benchmark_name`
- `metadata.source_url`
- `metadata.reference_path`, unless the benchmark has a separate official scorer integration
- `metadata.slices` including `public` plus benchmark-specific slices
- an existing `image_path`
- a matching image `sha256`, when the manifest records one
- a matching `metadata.text_sha256`, when the manifest records one
Audit it before running:
```bash
ocrtech audit-public-benchmark data/public/omnidocbench/manifests/omnidocbench-eval.jsonl \
--benchmark opendatalab/OmniDocBench \
--out outputs/public-omnidocbench-audit
```
This audit intentionally fails if provenance, references, images, or recorded
hashes are missing or stale. Do not use `--allow-missing-reference-paths` for
claim runs unless an official external scorer is wired and documented in the
experiment report.
Portfolio review for SOTA claims should use:
```bash
ocrtech review-claim \
--experiment outputs/internal-experiment/experiment.json \
--experiment outputs/public-omnidocbench/experiment.json \
--config configs/claim-review-sota-v1.json \
--model-config outputs/figure-caption-table-lexicon-v1/model-card.json \
--out outputs/claim-review-sota-v1
```
The config requires internal Nepali-English evidence plus public benchmark
evidence. It should fail until those public experiments actually exist.
Claim-grade validation configs can also set `min_pair_win_rate` on a gate. This
is intentionally different from the reported `win_rate`: `win_rate` is computed
over non-tied pairs, while `min_pair_win_rate` is checked as wins divided by all
paired samples. That distinction matters for document OCR claims. A result with
one win and many ties can have `win_rate = 1.0`, but it has low pair coverage
and should not satisfy a broad "better than baseline" claim unless the gate is
explicitly a no-regression/tie-acceptable gate.
The public OmniDocBench-vs-Surya validation gate is:
```bash
ocrtech validate-claim \
--benchmark-report outputs/experiment-public-omnidocbench-smoke-v1/benchmark/report.json \
--config configs/validation-public-omnidocbench-surya.json \
--eval-manifest data/public/omnidocbench-smoke/manifests/omnidocbench-eval.jsonl \
--candidate-model-config runs/recognizer/package/model-card.json \
--fallback-model-config runs/vlm-fallback/package/model-card.json \
--out outputs/validation-public-omnidocbench-smoke-v1
```
It requires at least `100` public document-parsing samples and at least `40`
table samples, then gates candidate-vs-Surya CER, table cell F1, structure-only
TEDS, and text-sensitive TEDS. A 5-page smoke run is allowed as engineering
evidence, but it must remain `not_validated`.
The first audited manifest sized for this gate is:
```text
data/public/omnidocbench-table100/manifests/omnidocbench-eval.jsonl
```
Its audit artifact is
`outputs/public-omnidocbench-table100-audit/public-benchmark-audit.json`; the
audit passed with `100` public/document/table English samples. The corresponding
long benchmark run is tracked under
`outputs/experiment-public-omnidocbench-table100-width-guard-v4/`. Treat that
run as incomplete until `benchmark/report.json`, `summary/summary.json`,
`validation/validation.json`, and `table-cell-analysis/table-cell-analysis.json`
exist.
The completed width-guard-v4 Table100 run is an instructive non-passing result:
- rows: `200/200`, with one shared `300s` timeout for candidate and Surya
- paired scored samples: `99`
- candidate improves public table metrics over Surya:
`table_cell_f1` `0.484364` vs `0.406694`,
`table_teds_structure` `0.720160` vs `0.687550`,
`table_teds_text` `0.565284` vs `0.543987`
- candidate does not improve OCR CER:
`0.613201` vs Surya `0.612623`
- validation status: `not_validated`
For long-run status files, distinguish package readiness from claim readiness.
`evidence_package_ready` means the expected rows and final report artifacts are
present. `paper_claim_ready` must only be true when the validation report itself
passes. A completed benchmark package with `validation.claim_status =
not_validated` is useful evidence, but not a publishable SOTA claim.
Long public runs should use benchmark checkpointing and per-page timeouts:
```bash
ocrtech benchmark ... \
--inputs-from-manifest data/eval/manifest-with-refs.jsonl \
--resume-existing \
--sample-timeout-seconds 300
```
`--inputs-from-manifest` loads authoritative image paths from the manifest and
defaults `--eval-manifest` to the same file when it is not supplied. This avoids
shell-generated input lists drifting away from the manifest that contains
reference paths and slice labels. `--resume-existing` scores existing
`document.json` outputs and only runs OCR for missing pages.
`--sample-timeout-seconds` runs each OCR call behind a hard process watchdog on
platforms that support `fork`; a stalled backend is terminated and recorded as a
page-level `TimeoutError`, then the benchmark continues. Platforms without
`fork` fall back to the Python signal timer. A timeout is still a
candidate/baseline failure for validation; it is not a pass-through.
For timeout runs, the benchmark now constructs the OCR engine inside the
watchdog child process for each page. This avoids inheriting already-initialized
PaddleOCR/PaddlePaddle C++ runtime state across `fork`, which produced
`exit_code=-11` worker crashes in the first line-level Paddle composite run.
Table scoring matches each reference table to the best unused predicted table
before averaging table metrics. This matters on public document pages where an
engine may emit extra table fragments before the main table. Positional
table-index matching is too brittle for claim evidence.
When the evaluator changes, do not rerun slow OCR just to update metrics. Use:
```bash
ocrtech rescore-benchmark \
--benchmark-report outputs/experiment-public-omnidocbench-smoke-v1/benchmark/report.json \
--eval-manifest data/public/omnidocbench-smoke/manifests/omnidocbench-eval.jsonl \
--out outputs/experiment-public-omnidocbench-smoke-v2-rescored/benchmark
```
The rescore command copies the existing `document.json`, Markdown, table, and
figure outputs into the new benchmark directory so downstream diagnostics such
as `analyze-table-cells` remain usable.
When evaluating table postprocessors, do not rely only on separate candidate and
baseline OCR runs. Independent OCR/layout calls can introduce run-to-run variance
that makes a small table postprocessor look better or worse than it is. Use a
fixed-source derived comparison to isolate the postprocessor:
```bash
ocrtech derive-table-postprocess-benchmark \
--benchmark-report outputs/experiment-public-omnidocbench-table40-normalize-rows-v3/benchmark/report.json \
--source-baseline surya \
--candidate-baseline candidate \
--eval-manifest data/public/omnidocbench-table40/manifests/omnidocbench-eval.jsonl \
--out outputs/experiment-public-omnidocbench-table40-derived-postprocess-v1/benchmark
```
This command copies the chosen source baseline outputs, applies the current
reference-free table postprocessors to the copied candidate `document.json`, and
rescored both systems against the same references. The result is valid evidence
for the postprocessor contribution only. It is not an end-to-end OCR model card
and cannot support a broad OCR SOTA claim by itself.
The current normalized public candidate is
`ocrtech-surya-tesseract-figure-table-lexicon-normalize-v2`. Its table-grid
normalization is runtime-only and reference-free: empty columns, separator-only
columns, and exact duplicate adjacent columns are removed. On the 5-page
OmniDocBench smoke it improves table metrics over Surya, but the run remains
`not_validated` because the public sample count, table sample count, CER margin,
and bootstrap confidence requirements are not met.
## Reference Format
Text-only reference:
```text
page.png.ref.txt
```
Structured reference:
```json
{
"text": "नेपाल सरकार",
"reading_order": ["p0-l0", "p0-l1", "p0-l2"],
"tables": [
[
["नाम", "उमेर", "जिल्ला"],
["राम", "३२", "काठमाडौं"]
]
],
"metadata": {
"source": "hand_labeled"
}
}
```
For manually reviewed Gorkhapatra language pages, `reading_order` may also use
structured objects. The scorer uses each object's `id` for ordering and keeps
the `text` as reviewer-visible reference context:
```json
{
"reading_order": [
{"id": "p0-l0", "type": "heading", "text": "मगर भाषा पृष्ठ"},
{"id": "p0-l1", "type": "paragraph", "text": "मुख्य समाचार"}
]
}
```
Complex table references can also use cell objects with spans:
```json
{
"tables": [
{
"cells": [
{"row": 0, "col": 0, "text": "नाम", "rowspan": 1, "colspan": 2},
{"row": 1, "col": 0, "text": "राम"},
{"row": 1, "col": 1, "text": "१२"}
]
}
]
}
```
The nested-list grid form remains supported. Span-aware references make `table_exact_match` and `table_structure_similarity` sensitive to missing rowspans/colspans even when flattened table text is correct.
The candidate parser can apply a conservative repeated-header span postprocessor:
adjacent identical cells in the first header rows are collapsed into a single
`colspan` cell only when the following row has distinct child columns. This lets
`document.json` and exported table HTML represent the simple merged-header
pattern in `complex-table-ledger`. Treat this as a heuristic parser repair until
it is validated on real scanned tables with varied merged cells and rowspans.
Supported structured metrics:
- `quality_score`
- `cer`
- `wer`
- `text_cer`
- `text_wer`
- `document_reconstruction_cer`
- `document_reconstruction_wer`
- `document_reconstruction_text_similarity`
- `document_reconstruction_similarity`
- `figure_detection_precision`
- `figure_detection_recall`
- `figure_detection_f1`
- `figure_caption_cer`
- `reading_order_pair_accuracy`
- `table_cell_precision`
- `table_cell_recall`
- `table_cell_f1`
- `table_exact_match`
- `table_structure_similarity`
- `table_teds_structure`
- `table_teds_text`
`document_reconstruction_text_similarity` is `1 - CER` over rendered Markdown
only. `document_reconstruction_similarity` is stricter: when the reference also
declares reading order, tables, or figures, the reconstruction score is capped
by those structure metrics. This prevents a parser from receiving full document
reconstruction credit by emitting Markdown-shaped text while failing to populate
structured `document.json` tables or figures.
## Validation Config
Example:
```json
{
"candidate": "ours",
"min_samples": 300,
"required_slices": {
"nepali": 100,
"english": 100,
"multi_page": 25,
"table": 50,
"reading_order": 50,
"figure": 30,
"scan": 50,
"phone_photo": 50
},
"bootstrap_iterations": 1000,
"bootstrap_confidence": 0.95,
"require_full_eval_manifest_coverage": true,
"gates": [
{
"baseline": "tesseract",
"metric": "cer",
"direction": "lower",
"slice": "nepali",
"min_pairs": 50,
"min_relative_improvement": 0.05,
"max_p_value": 0.05
},
{
"baseline": "stock-paddle",
"metric": "cer",
"direction": "lower",
"slice": "english",
"min_absolute_margin": 0.0
},
{
"baseline": "glm-ocr",
"metric": "table_cell_f1",
"direction": "higher",
"slice": "table",
"min_absolute_margin": 0.01
},
{
"baseline": "surya",
"metric": "cer",
"direction": "lower",
"slices": ["hard_eval", "table"],
"min_pairs": 8,
"min_absolute_margin": 0.02
},
{
"baseline": "paddleocr-vl",
"metric": "reading_order_pair_accuracy",
"direction": "higher",
"slice": "reading_order",
"min_absolute_margin": 0.01
}
]
}
```
Use `slice` for a single required slice. Use `slices` when a gate must apply to the intersection of multiple labels, for example `["hard_eval", "table"]` for table samples inside a hard-eval pocket. Do not approximate this with separate `hard_eval` and `table` gates; that allows unrelated samples to satisfy the table gate.
For claim-grade runs, set `require_full_eval_manifest_coverage` to `true`.
When enabled, `validate-claim` fails unless the candidate and every baseline
named by the configured gates have a successful row for every sample in
`--eval-manifest`. This prevents a validation report from passing on an easy
subset while silently omitting hard pages from the held-out manifest.
Run:
```bash
ocrtech preflight-claim \
--eval-manifest data/manifests/eval.jsonl \
--references-dir data/references \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--candidate-model-config runs/recognizer/package/model-card.json \
--validation-config validation.json \
--train-manifest data/manifests/train.jsonl \
--out outputs/preflight
ocrtech benchmark \
--eval-manifest data/manifests/eval.jsonl \
--references-dir data/references \
--baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--out outputs/benchmark \
$(cut -f4 data/manifests/eval.inputs)
ocrtech validate-claim \
--benchmark-report outputs/benchmark/report.json \
--config validation.json \
--eval-manifest data/manifests/eval.jsonl \
--train-manifest data/manifests/train.jsonl \
--out outputs/validation
```
`preflight-claim` should be the first command in a claim-grade workflow. If it fails, the benchmark run is not ready.
For explicit `--validation-config` runs, preflight now checks the config against
the requested baselines and eval manifest before any OCR starts. It fails when
the candidate label is absent from `--baselines`, a gate baseline is not
scheduled, `min_samples` exceeds the manifest size, required slices are missing,
or a gate's `min_pairs` exceeds the number of manifest rows matching the gate
slice intersection. Omitting `--validation-config` is only a warning; it is not
claim-grade evidence.
For hybrid runs with `--fallback-model-config`, preflight also checks that the
candidate model card declares the same fallback under
`provenance.fallback_model_card`. This catches a wrong runtime fallback flag
before a benchmark produces evidence for a different composite engine than the
one described by the candidate model card.
If the candidate model uses `quality_select_composite`, `preflight-claim` now checks that the model card contains calibration provenance and that the calibration eval manifest hash is different from the final claim eval manifest hash. Tuning on the same eval pack invalidates the claim path.
`validate-claim` exits with `0` only when the configured claim is validated. It exits with `1` when evidence is missing or a gate fails.
Each gate now reports:
- paired sample count
- win / loss / tie counts
- win rate on non-tied pairs
- bootstrap CI for mean paired improvement
- optional one-sided sign-test p-value
For claim-grade runs, use `min_pairs` and `max_p_value` instead of relying only on a favorable mean. A narrow win on a handful of pages is not SOTA evidence.
For a single reproducible command that runs benchmark and validation together, use `run-experiment`; see [EXPERIMENTS.md](EXPERIMENTS.md).
For the final portfolio-level claim review across the internal pack and public suites, use `review-claim`; see [CLAIM_REVIEW.md](CLAIM_REVIEW.md).
For model artifact provenance, see [MODELS.md](MODELS.md). Do not use an implicit `auto` engine for SOTA claims.
## Table Cell Analysis
Aggregate `table_cell_f1` is not enough to explain whether a table-oriented candidate is actually improving table OCR. Use `analyze-table-cells` after a benchmark has exported per-system `document.json` files:
```bash
ocrtech analyze-table-cells \
--benchmark-report outputs/experiment-table-cell-refine-empty-hard15-v1/benchmark/report.json \
--eval-manifest data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
--out outputs/table-cell-analysis-refine-empty-hard15-v1 \
--candidate candidate \
--baseline surya \
--baseline tesseract
```
The command requires:
- A benchmark report from `benchmark` or `run-experiment`.
- Exported benchmark documents at `benchmark/<system>/<input_stem>/document.json`.
- An eval manifest whose references contain structured `tables`.
It writes:
- `table-cell-analysis.json`
- `table-cell-analysis.md`
The report includes reference table count, predicted table count, duplicate overlapping table count, reference cell count, exact cell count, exact-cell rate, missing-cell count, mean cell CER, paired candidate-vs-baseline wins/losses/ties, and the highest-error candidate cells. This is the right artifact for identifying whether a table refiner is adding headroom, damaging already-correct cells, or producing structurally duplicated tables.
Interpret raw full-page Tesseract table results carefully. In the current v1 engine, Tesseract can be strong on exact image crops but still report zero structured table cells in full-page document parsing because it does not emit the table structure expected by `document.json`.
For the rendered hard15 table pages, reference `text` is the page title, while table contents live in structured `tables`. Do not mix these contracts: page-level `cer` answers whether the surrounding page text/title was recognized, and `table_cell_f1` plus cell exact analysis answers whether the table content was extracted. A benchmark change that inserts table grids into `Document.text` without also regenerating references changes the metric definition and is invalid for this pack.
The current `validation-hard15-surya.json` table-cell gate can fail even with a higher mean table-cell F1 when the bootstrap confidence interval includes zero. The figure-caption/table-lexicon v3 run is the concrete example: candidate `1.000000` vs Surya `0.979167`, but only 2 table pages improve and 6 tie, so the CI low is `0.000000`. Treat that as evidence that the table slice is too small for a strong claim. Do not remove the CI gate after observing the result; build a larger held-out table pack.
`configs/validation-table-ledger-surya.json` is the first targeted table-only validation config. It requires at least 40 table samples, gates `table_cell_f1` and `table_exact_match` against Surya with bootstrap CI above zero, and guards `table_structure_similarity` for non-regression. The v2 table-ledger pack (`60` samples, `720` cells) passes this config:
- `table_cell_f1`: candidate `1.000000`, Surya `0.979167`, CI low `0.012500`
- `table_exact_match`: candidate `1.000000`, Surya `0.750000`, CI low `0.150000`
- `table_structure_similarity`: candidate `1.000000`, Surya `1.000000`, non-regression guard passes
`table_structure_similarity` is a deterministic table-structure score over row count, column count, occupied cell coordinates, and rowspans/colspans when structured cell references are present. It is still not full HTML tree-edit TEDS. In the table-ledger v2 run it correctly shows no structure improvement over Surya because both systems already recover the simple grid structure perfectly. The candidate's validated gain is table cell-content accuracy with structure non-regression.
`table_teds_structure` and `table_teds_text` are deterministic TEDS-style ordered tree-edit scores over a table DOM made of `table`, `tr`, and `td` nodes. Structure mode compares row/cell tree shape plus `rowspan`/`colspan`; text mode also includes normalized cell text in `td` node labels. These are closer to paper-grade table metrics than the older token score, but they are still internal deterministic metrics, not a substitute for running an accepted public benchmark implementation when making broad SOTA claims.
After the span-inference postprocessor and tree metrics, the generated complex-table pack shows a structure gain over Surya (`table_structure_similarity` `1.000000` vs `0.875000`, `table_teds_structure` `1.000000` vs `0.904762`, `table_teds_text` `0.996250` vs `0.891667`, `table_exact_match` `0.925000` vs `0.000000`). This is evidence for the generated repeated-header pattern only. It does not replace a public or real scanned-table benchmark.
The public OmniDocBench smoke now also has a row-fragment normalization
candidate. It improves public table metrics over Surya on the same 5-page smoke
(`table_cell_f1` `0.564335` vs `0.296264`, `table_teds_structure` `0.942836`
vs `0.740128`, `table_teds_text` `0.675633` vs `0.526327`) and fixes one
`16x5` fragmented table into the expected `3x5` shape. It is still
`not_validated`: the sample count is far below the configured public gates,
bootstrap CI lower bounds are `0.000000` on table metrics, and text CER is
slightly worse than Surya on the smoke (`0.481629` vs `0.480531`).
The same sparse-row normalizer now recognizes numbered list markers such as
`1)`, `2.`, `(a)`, and Devanagari-digit variants. This is still guarded by the
same single-cell continuation-row rule and should be treated as a parser
postprocessor until rerun on the public table gate.
The larger public OmniDocBench `table40` run is more important. It reaches the
configured table-pair count (`40` public table pages) but still fails validation:
`table_cell_f1` improves in mean (`0.323122` vs `0.280062`) while losing more
pages than it wins (`7` wins, `9` losses, `24` ties), and the bootstrap CI low is
negative (`-0.019152`). `table_teds_structure` also has a positive mean
(`0.520959` vs `0.503603`) but CI low is negative (`-0.038381`). `table_teds_text`
does not clear the margin (`0.003609` improvement). The document CER gate is
also not met: only `40/100` required public document pages, and CER improvement
is only `0.000634` against a `0.005` margin.
The width-guarded v4 rerun improves the public table40 evidence but still does
not validate the full claim. Skipping table-grid normalization on tables wider
than `12` columns makes the table-cell F1 gate pass (`0.339319` vs `0.280062`,
CI low `0.001433`) and the table TEDS-structure gate pass (`0.545375` vs
`0.503603`, CI low `0.004790`). The text-sensitive table gate still fails:
`table_teds_text` improves in mean (`0.386013` vs `0.358057`) but CI low is
`-0.000220`. The public document CER gate also still fails because the slice has
only `40/100` required public document pages, despite a mean CER improvement of
`0.006096`.
On `2026-06-14`, the `table100` public run was launched on `k2` using
`scripts/run_public_table100_claim.py` and the v4 width-guard model card. The
first all-or-nothing launch stalled after three completed candidate rows. We
restarted with an isolated per-page runner, `--resume-existing`, and
`--sample-timeout-seconds 300`, preserving completed outputs and forcing future
stalls to become explicit error rows. The first measured timeout row was
`omnidocbench-00216-page-dca64e05-1ce6-49ee-857a-9b5d05a87357`. Do not cite this
run as a result until the validation artifact exists and has been inspected.
This validates a targeted table-ledger extraction claim, not full document SOTA. Full SOTA validation still needs broad document coverage, real scanned tables, merged-cell/header cases, true TEDS/TEDS-IOU, and public benchmark evidence.
## Recognizer Hard-Failure Gate
The failed AWS G6 PaddleOCR recognizer produced a normalized `50`-sample
hard-eval slice:
```text
data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl
```
Use this slice as a regression gate for the next trained recognizer branch. It
must remain held out; do not train on it.
Validation config:
```text
configs/validation-paddle-hard50-tesseract.json
```
Run it after benchmarking a candidate recognizer against Tesseract:
```bash
ocrtech validate-claim \
--benchmark-report outputs/next-recognizer-hard50/benchmark/report.json \
--config configs/validation-paddle-hard50-tesseract.json \
--eval-manifest data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl \
--out outputs/validation-next-recognizer-hard50
```
This gate is intentionally framed as failure recovery:
- all `50` hard rows must be present
- the `38` English synthetic line-crop failures must not regress versus
Tesseract on CER or WER
- the `12` Nepali hard failures must improve WER over Tesseract by at least
`0.05`
Passing this gate alone is not a SOTA claim. It is a recognizer branch admission
test before rerunning the full mixed OCR eval, rendered document pack, and public
benchmark gates.
The first line-level Tesseract/Paddle composite run exposed a benchmark-lifecycle
bug, not a model result: every candidate row crashed with `exit_code=-11` because
PaddleOCR was initialized in the parent process and then inherited by forked
timeout workers. After changing timeout mode to initialize engines inside the
watchdog worker, the rerun completed all `100` rows:
```text
outputs/experiment-line-script-paddle-hard50-v2/report.json
outputs/summary-line-script-paddle-hard50-v2/summary.md
outputs/validation-line-script-paddle-hard50-v2/validation.md
```
Validation status: `not_validated`.
- hard-50 CER: candidate `0.276489`, Tesseract `0.120106`
- English line-crop CER: candidate `0.060429`, Tesseract `0.060429`
- English line-crop WER: candidate `0.132259`, Tesseract `0.132259`
- Nepali hard CER: candidate `0.960679`, Tesseract `0.309083`
- Nepali hard WER: candidate `1.000000`, Tesseract `0.711607`
Interpretation: line-level routing can preserve English by leaving English lines
on the Tesseract primary path, but the trained PaddleOCR secondary recognizer is
not admissible for Devanagari overrides.
## Leakage Rules
Fatal leakage:
- same `sample_id`
- same image SHA-256
- same image path
- same image+text sample signature
Warning only:
- same text SHA-256
Text-only overlap is not fatal because short OCR labels repeat naturally across real corpora. It is still reported because repeated long labels may indicate bad dataset splitting.
## External Benchmarks
Use public suites for broad claims and the internal Nepali/English pack for the actual target domain:
- OmniDocBench for general document parsing.
- olmOCR-bench for PDF-to-Markdown structure preservation.
- OCRBench / OCRBench v2 for VLM OCR coverage.
- Internal Nepali + English pages for Devanagari OCR, mixed-script pages, government forms, tables, captions, and noisy scans.
For table claims, cell exact match, `table_cell_f1`, and span-aware `table_structure_similarity` are v1 metrics. Add true TEDS/TEDS-IOU before claiming leadership on complex merged-cell tables.
## Internal Eval Pack
See [EVAL_PACK.md](EVAL_PACK.md). The internal pack is necessary because public OCR/document benchmarks do not sufficiently cover Nepali, mixed Devanagari-English pages, and local document layouts.
## Dataset Splitting
See [DATASETS.md](DATASETS.md). Do not manually split data when making benchmark claims; use `split-manifest` so duplicate sample signatures cannot cross train/eval boundaries.