Download ocr-tech-mps-integration/docs/VALIDATION.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 31.8 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/VALIDATION.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech-mps-integration/docs/VALIDATION.md
-
curl -L -o VALIDATION.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/VALIDATION.md
Validation Design
The claim target is not "we ran OCR and it looked good." The claim target is:
ocrtech is better than named baselines on held-out Nepali + English OCR and document-structure extraction, under measured gates.
Until the validation report says validated, do not claim SOTA.
Claim-grade validation.json files record provenance for the benchmark report,
validation config, eval manifest, train manifests, candidate model card, and
fallback model card used by validate-claim. review-claim cross-checks
those hashes against experiment.json, so a validation artifact from a
different benchmark, eval split, validation config, model card, fallback card,
or training-manifest set cannot be reused as portfolio evidence.
The SOTA review profile is paired with these claim-grade validation configs:
configs/validation-internal-sota-v1.jsonconfigs/validation-public-omnidocbench-sota-v1.jsonconfigs/validation-public-real5-sota-v1.json
Each one covers the required_validation_gates declared in
configs/claim-review-sota-v1.json. Use the slot-specific config when running
run-experiment or validate-claim; otherwise portfolio review may have
metrics and baselines but still lack the passed gates required for the paper
claim.
The built-in fallback validation config is intentionally broader than a plain
OCR smoke check. If validate-claim is called without --config, it now gates
Nepali CER/WER, English non-regression, table cell and TEDS-text quality,
reading order, document reconstruction, and figure detection/caption quality.
That default remains smaller than the SOTA configs above, but it should not
allow a "document OCR" claim to pass on text-only evidence.
Evidence Required
A defensible run needs all of this:
- A claim-ready image eval pack from
ocrtech create-eval-pack, plus public benchmark runs. - A candidate model card from
ocrtech create-model-cardand a passingocrtech audit-modelreport. - Training data prepared with
audit-manifestandsplit-manifest. - Held-out eval manifest that was never used for recognizer, corrector, threshold, or prompt tuning.
- Image/PDF eval inputs normalized to OCR-supported file suffixes before any claim-grade PaddleOCR/Tesseract/Surya/VLM comparison.
- For routed or thresholded candidates, a separate calibration experiment artifact that records how the threshold was chosen.
- Train manifests for leakage checks.
- Paired benchmark outputs for every baseline on the same input files.
- Per-sample metrics, not only aggregate averages.
- For
glm-ocrandpaddleocr-vl, a real runnable command hook that emits Markdown ordocument.md, not a placeholder env var. - Slice coverage for Nepali, English, multi-page documents, tables, reading order, figures/charts, forms, scans, photos, skew, blur, compression, and low contrast.
- Gates that name the baseline, metric, slice, direction, required margin, and confidence requirement.
- Paired win/loss evidence, a minimum paired win-rate requirement when the claim needs broad per-document improvement, and a nonparametric significance check when the claim requires it.
- Failure-case report, not just the winning average.
Public Benchmark Gate
Generated eval packs are development evidence. They are not enough for SOTA
wording. Before a paper claim can say "state of the art," at least one public
document parsing benchmark must be converted into an auditable manifest and run
as a normal ocrtech run-experiment.
Known public benchmark registry:
ocrtech list-public-benchmarks
Current registry entries:
opendatalab/OmniDocBench: real-world document parsing benchmark, dataset pagehttps://huggingface.co/datasets/opendatalab/OmniDocBenchallenai/olmOCR-bench: PDF-to-Markdown benchmark with unit-test style assertions, dataset pagehttps://huggingface.co/datasets/allenai/olmOCR-benchPaddlePaddle/Real5-OmniDocBench: real-world physical reconstruction benchmark with scanning, warping, screen-photo, illumination, and skew scenarios, dataset pagehttps://huggingface.co/datasets/PaddlePaddle/Real5-OmniDocBench
The first implemented public converter is:
ocrtech prepare-public-benchmark opendatalab/OmniDocBench \
--out data/public/omnidocbench \
--limit 100 \
--language english \
--require-tables
The converter was written only after checking the actual dataset structure:
OmniDocBench.json contains page_info, page_info.page_attribute, and
layout_dets; image files resolve as images/{page_info.image_path}; table
ground truth is stored in table-block html.
Every public benchmark manifest row must include:
metadata.public_benchmark = truemetadata.benchmark_namemetadata.source_urlmetadata.reference_path, unless the benchmark has a separate official scorer integrationmetadata.slicesincludingpublicplus benchmark-specific slices- an existing
image_path - a matching image
sha256, when the manifest records one - a matching
metadata.text_sha256, when the manifest records one
Audit it before running:
ocrtech audit-public-benchmark data/public/omnidocbench/manifests/omnidocbench-eval.jsonl \
--benchmark opendatalab/OmniDocBench \
--out outputs/public-omnidocbench-audit
This audit intentionally fails if provenance, references, images, or recorded
hashes are missing or stale. Do not use --allow-missing-reference-paths for
claim runs unless an official external scorer is wired and documented in the
experiment report.
Portfolio review for SOTA claims should use:
ocrtech review-claim \
--experiment outputs/internal-experiment/experiment.json \
--experiment outputs/public-omnidocbench/experiment.json \
--config configs/claim-review-sota-v1.json \
--model-config outputs/figure-caption-table-lexicon-v1/model-card.json \
--out outputs/claim-review-sota-v1
The config requires internal Nepali-English evidence plus public benchmark evidence. It should fail until those public experiments actually exist.
Claim-grade validation configs can also set min_pair_win_rate on a gate. This
is intentionally different from the reported win_rate: win_rate is computed
over non-tied pairs, while min_pair_win_rate is checked as wins divided by all
paired samples. That distinction matters for document OCR claims. A result with
one win and many ties can have win_rate = 1.0, but it has low pair coverage
and should not satisfy a broad "better than baseline" claim unless the gate is
explicitly a no-regression/tie-acceptable gate.
The public OmniDocBench-vs-Surya validation gate is:
ocrtech validate-claim \
--benchmark-report outputs/experiment-public-omnidocbench-smoke-v1/benchmark/report.json \
--config configs/validation-public-omnidocbench-surya.json \
--eval-manifest data/public/omnidocbench-smoke/manifests/omnidocbench-eval.jsonl \
--candidate-model-config runs/recognizer/package/model-card.json \
--fallback-model-config runs/vlm-fallback/package/model-card.json \
--out outputs/validation-public-omnidocbench-smoke-v1
It requires at least 100 public document-parsing samples and at least 40
table samples, then gates candidate-vs-Surya CER, table cell F1, structure-only
TEDS, and text-sensitive TEDS. A 5-page smoke run is allowed as engineering
evidence, but it must remain not_validated.
The first audited manifest sized for this gate is:
data/public/omnidocbench-table100/manifests/omnidocbench-eval.jsonl
Its audit artifact is
outputs/public-omnidocbench-table100-audit/public-benchmark-audit.json; the
audit passed with 100 public/document/table English samples. The corresponding
long benchmark run is tracked under
outputs/experiment-public-omnidocbench-table100-width-guard-v4/. Treat that
run as incomplete until benchmark/report.json, summary/summary.json,
validation/validation.json, and table-cell-analysis/table-cell-analysis.json
exist.
The completed width-guard-v4 Table100 run is an instructive non-passing result:
- rows:
200/200, with one shared300stimeout for candidate and Surya - paired scored samples:
99 - candidate improves public table metrics over Surya:
table_cell_f10.484364vs0.406694,table_teds_structure0.720160vs0.687550,table_teds_text0.565284vs0.543987 - candidate does not improve OCR CER:
0.613201vs Surya0.612623 - validation status:
not_validated
For long-run status files, distinguish package readiness from claim readiness.
evidence_package_ready means the expected rows and final report artifacts are
present. paper_claim_ready must only be true when the validation report itself
passes. A completed benchmark package with validation.claim_status = not_validated is useful evidence, but not a publishable SOTA claim.
Long public runs should use benchmark checkpointing and per-page timeouts:
ocrtech benchmark ... \
--inputs-from-manifest data/eval/manifest-with-refs.jsonl \
--resume-existing \
--sample-timeout-seconds 300
--inputs-from-manifest loads authoritative image paths from the manifest and
defaults --eval-manifest to the same file when it is not supplied. This avoids
shell-generated input lists drifting away from the manifest that contains
reference paths and slice labels. --resume-existing scores existing
document.json outputs and only runs OCR for missing pages.
--sample-timeout-seconds runs each OCR call behind a hard process watchdog on
platforms that support fork; a stalled backend is terminated and recorded as a
page-level TimeoutError, then the benchmark continues. Platforms without
fork fall back to the Python signal timer. A timeout is still a
candidate/baseline failure for validation; it is not a pass-through.
For timeout runs, the benchmark now constructs the OCR engine inside the
watchdog child process for each page. This avoids inheriting already-initialized
PaddleOCR/PaddlePaddle C++ runtime state across fork, which produced
exit_code=-11 worker crashes in the first line-level Paddle composite run.
Table scoring matches each reference table to the best unused predicted table before averaging table metrics. This matters on public document pages where an engine may emit extra table fragments before the main table. Positional table-index matching is too brittle for claim evidence.
When the evaluator changes, do not rerun slow OCR just to update metrics. Use:
ocrtech rescore-benchmark \
--benchmark-report outputs/experiment-public-omnidocbench-smoke-v1/benchmark/report.json \
--eval-manifest data/public/omnidocbench-smoke/manifests/omnidocbench-eval.jsonl \
--out outputs/experiment-public-omnidocbench-smoke-v2-rescored/benchmark
The rescore command copies the existing document.json, Markdown, table, and
figure outputs into the new benchmark directory so downstream diagnostics such
as analyze-table-cells remain usable.
When evaluating table postprocessors, do not rely only on separate candidate and baseline OCR runs. Independent OCR/layout calls can introduce run-to-run variance that makes a small table postprocessor look better or worse than it is. Use a fixed-source derived comparison to isolate the postprocessor:
ocrtech derive-table-postprocess-benchmark \
--benchmark-report outputs/experiment-public-omnidocbench-table40-normalize-rows-v3/benchmark/report.json \
--source-baseline surya \
--candidate-baseline candidate \
--eval-manifest data/public/omnidocbench-table40/manifests/omnidocbench-eval.jsonl \
--out outputs/experiment-public-omnidocbench-table40-derived-postprocess-v1/benchmark
This command copies the chosen source baseline outputs, applies the current
reference-free table postprocessors to the copied candidate document.json, and
rescored both systems against the same references. The result is valid evidence
for the postprocessor contribution only. It is not an end-to-end OCR model card
and cannot support a broad OCR SOTA claim by itself.
The current normalized public candidate is
ocrtech-surya-tesseract-figure-table-lexicon-normalize-v2. Its table-grid
normalization is runtime-only and reference-free: empty columns, separator-only
columns, and exact duplicate adjacent columns are removed. On the 5-page
OmniDocBench smoke it improves table metrics over Surya, but the run remains
not_validated because the public sample count, table sample count, CER margin,
and bootstrap confidence requirements are not met.
Reference Format
Text-only reference:
page.png.ref.txt
Structured reference:
{
"text": "नेपाल सरकार",
"reading_order": ["p0-l0", "p0-l1", "p0-l2"],
"tables": [
[
["नाम", "उमेर", "जिल्ला"],
["राम", "३२", "काठमाडौं"]
]
],
"metadata": {
"source": "hand_labeled"
}
}
For manually reviewed Gorkhapatra language pages, reading_order may also use
structured objects. The scorer uses each object's id for ordering and keeps
the text as reviewer-visible reference context:
{
"reading_order": [
{"id": "p0-l0", "type": "heading", "text": "मगर भाषा पृष्ठ"},
{"id": "p0-l1", "type": "paragraph", "text": "मुख्य समाचार"}
]
}
Complex table references can also use cell objects with spans:
{
"tables": [
{
"cells": [
{"row": 0, "col": 0, "text": "नाम", "rowspan": 1, "colspan": 2},
{"row": 1, "col": 0, "text": "राम"},
{"row": 1, "col": 1, "text": "१२"}
]
}
]
}
The nested-list grid form remains supported. Span-aware references make table_exact_match and table_structure_similarity sensitive to missing rowspans/colspans even when flattened table text is correct.
The candidate parser can apply a conservative repeated-header span postprocessor:
adjacent identical cells in the first header rows are collapsed into a single
colspan cell only when the following row has distinct child columns. This lets
document.json and exported table HTML represent the simple merged-header
pattern in complex-table-ledger. Treat this as a heuristic parser repair until
it is validated on real scanned tables with varied merged cells and rowspans.
Supported structured metrics:
quality_scorecerwertext_certext_werdocument_reconstruction_cerdocument_reconstruction_werdocument_reconstruction_text_similaritydocument_reconstruction_similarityfigure_detection_precisionfigure_detection_recallfigure_detection_f1figure_caption_cerreading_order_pair_accuracytable_cell_precisiontable_cell_recalltable_cell_f1table_exact_matchtable_structure_similaritytable_teds_structuretable_teds_text
document_reconstruction_text_similarity is 1 - CER over rendered Markdown
only. document_reconstruction_similarity is stricter: when the reference also
declares reading order, tables, or figures, the reconstruction score is capped
by those structure metrics. This prevents a parser from receiving full document
reconstruction credit by emitting Markdown-shaped text while failing to populate
structured document.json tables or figures.
Validation Config
Example:
{
"candidate": "ours",
"min_samples": 300,
"required_slices": {
"nepali": 100,
"english": 100,
"multi_page": 25,
"table": 50,
"reading_order": 50,
"figure": 30,
"scan": 50,
"phone_photo": 50
},
"bootstrap_iterations": 1000,
"bootstrap_confidence": 0.95,
"require_full_eval_manifest_coverage": true,
"gates": [
{
"baseline": "tesseract",
"metric": "cer",
"direction": "lower",
"slice": "nepali",
"min_pairs": 50,
"min_relative_improvement": 0.05,
"max_p_value": 0.05
},
{
"baseline": "stock-paddle",
"metric": "cer",
"direction": "lower",
"slice": "english",
"min_absolute_margin": 0.0
},
{
"baseline": "glm-ocr",
"metric": "table_cell_f1",
"direction": "higher",
"slice": "table",
"min_absolute_margin": 0.01
},
{
"baseline": "surya",
"metric": "cer",
"direction": "lower",
"slices": ["hard_eval", "table"],
"min_pairs": 8,
"min_absolute_margin": 0.02
},
{
"baseline": "paddleocr-vl",
"metric": "reading_order_pair_accuracy",
"direction": "higher",
"slice": "reading_order",
"min_absolute_margin": 0.01
}
]
}
Use slice for a single required slice. Use slices when a gate must apply to the intersection of multiple labels, for example ["hard_eval", "table"] for table samples inside a hard-eval pocket. Do not approximate this with separate hard_eval and table gates; that allows unrelated samples to satisfy the table gate.
For claim-grade runs, set require_full_eval_manifest_coverage to true.
When enabled, validate-claim fails unless the candidate and every baseline
named by the configured gates have a successful row for every sample in
--eval-manifest. This prevents a validation report from passing on an easy
subset while silently omitting hard pages from the held-out manifest.
Run:
ocrtech preflight-claim \
--eval-manifest data/manifests/eval.jsonl \
--references-dir data/references \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--candidate-model-config runs/recognizer/package/model-card.json \
--validation-config validation.json \
--train-manifest data/manifests/train.jsonl \
--out outputs/preflight
ocrtech benchmark \
--eval-manifest data/manifests/eval.jsonl \
--references-dir data/references \
--baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--out outputs/benchmark \
$(cut -f4 data/manifests/eval.inputs)
ocrtech validate-claim \
--benchmark-report outputs/benchmark/report.json \
--config validation.json \
--eval-manifest data/manifests/eval.jsonl \
--train-manifest data/manifests/train.jsonl \
--out outputs/validation
preflight-claim should be the first command in a claim-grade workflow. If it fails, the benchmark run is not ready.
For explicit --validation-config runs, preflight now checks the config against
the requested baselines and eval manifest before any OCR starts. It fails when
the candidate label is absent from --baselines, a gate baseline is not
scheduled, min_samples exceeds the manifest size, required slices are missing,
or a gate's min_pairs exceeds the number of manifest rows matching the gate
slice intersection. Omitting --validation-config is only a warning; it is not
claim-grade evidence.
For hybrid runs with --fallback-model-config, preflight also checks that the
candidate model card declares the same fallback under
provenance.fallback_model_card. This catches a wrong runtime fallback flag
before a benchmark produces evidence for a different composite engine than the
one described by the candidate model card.
If the candidate model uses quality_select_composite, preflight-claim now checks that the model card contains calibration provenance and that the calibration eval manifest hash is different from the final claim eval manifest hash. Tuning on the same eval pack invalidates the claim path.
validate-claim exits with 0 only when the configured claim is validated. It exits with 1 when evidence is missing or a gate fails.
Each gate now reports:
- paired sample count
- win / loss / tie counts
- win rate on non-tied pairs
- bootstrap CI for mean paired improvement
- optional one-sided sign-test p-value
For claim-grade runs, use min_pairs and max_p_value instead of relying only on a favorable mean. A narrow win on a handful of pages is not SOTA evidence.
For a single reproducible command that runs benchmark and validation together, use run-experiment; see EXPERIMENTS.md.
For the final portfolio-level claim review across the internal pack and public suites, use review-claim; see CLAIM_REVIEW.md.
For model artifact provenance, see MODELS.md. Do not use an implicit auto engine for SOTA claims.
Table Cell Analysis
Aggregate table_cell_f1 is not enough to explain whether a table-oriented candidate is actually improving table OCR. Use analyze-table-cells after a benchmark has exported per-system document.json files:
ocrtech analyze-table-cells \
--benchmark-report outputs/experiment-table-cell-refine-empty-hard15-v1/benchmark/report.json \
--eval-manifest data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
--out outputs/table-cell-analysis-refine-empty-hard15-v1 \
--candidate candidate \
--baseline surya \
--baseline tesseract
The command requires:
- A benchmark report from
benchmarkorrun-experiment. - Exported benchmark documents at
benchmark/<system>/<input_stem>/document.json. - An eval manifest whose references contain structured
tables.
It writes:
table-cell-analysis.jsontable-cell-analysis.md
The report includes reference table count, predicted table count, duplicate overlapping table count, reference cell count, exact cell count, exact-cell rate, missing-cell count, mean cell CER, paired candidate-vs-baseline wins/losses/ties, and the highest-error candidate cells. This is the right artifact for identifying whether a table refiner is adding headroom, damaging already-correct cells, or producing structurally duplicated tables.
Interpret raw full-page Tesseract table results carefully. In the current v1 engine, Tesseract can be strong on exact image crops but still report zero structured table cells in full-page document parsing because it does not emit the table structure expected by document.json.
For the rendered hard15 table pages, reference text is the page title, while table contents live in structured tables. Do not mix these contracts: page-level cer answers whether the surrounding page text/title was recognized, and table_cell_f1 plus cell exact analysis answers whether the table content was extracted. A benchmark change that inserts table grids into Document.text without also regenerating references changes the metric definition and is invalid for this pack.
The current validation-hard15-surya.json table-cell gate can fail even with a higher mean table-cell F1 when the bootstrap confidence interval includes zero. The figure-caption/table-lexicon v3 run is the concrete example: candidate 1.000000 vs Surya 0.979167, but only 2 table pages improve and 6 tie, so the CI low is 0.000000. Treat that as evidence that the table slice is too small for a strong claim. Do not remove the CI gate after observing the result; build a larger held-out table pack.
configs/validation-table-ledger-surya.json is the first targeted table-only validation config. It requires at least 40 table samples, gates table_cell_f1 and table_exact_match against Surya with bootstrap CI above zero, and guards table_structure_similarity for non-regression. The v2 table-ledger pack (60 samples, 720 cells) passes this config:
table_cell_f1: candidate1.000000, Surya0.979167, CI low0.012500table_exact_match: candidate1.000000, Surya0.750000, CI low0.150000table_structure_similarity: candidate1.000000, Surya1.000000, non-regression guard passes
table_structure_similarity is a deterministic table-structure score over row count, column count, occupied cell coordinates, and rowspans/colspans when structured cell references are present. It is still not full HTML tree-edit TEDS. In the table-ledger v2 run it correctly shows no structure improvement over Surya because both systems already recover the simple grid structure perfectly. The candidate's validated gain is table cell-content accuracy with structure non-regression.
table_teds_structure and table_teds_text are deterministic TEDS-style ordered tree-edit scores over a table DOM made of table, tr, and td nodes. Structure mode compares row/cell tree shape plus rowspan/colspan; text mode also includes normalized cell text in td node labels. These are closer to paper-grade table metrics than the older token score, but they are still internal deterministic metrics, not a substitute for running an accepted public benchmark implementation when making broad SOTA claims.
After the span-inference postprocessor and tree metrics, the generated complex-table pack shows a structure gain over Surya (table_structure_similarity 1.000000 vs 0.875000, table_teds_structure 1.000000 vs 0.904762, table_teds_text 0.996250 vs 0.891667, table_exact_match 0.925000 vs 0.000000). This is evidence for the generated repeated-header pattern only. It does not replace a public or real scanned-table benchmark.
The public OmniDocBench smoke now also has a row-fragment normalization
candidate. It improves public table metrics over Surya on the same 5-page smoke
(table_cell_f1 0.564335 vs 0.296264, table_teds_structure 0.942836
vs 0.740128, table_teds_text 0.675633 vs 0.526327) and fixes one
16x5 fragmented table into the expected 3x5 shape. It is still
not_validated: the sample count is far below the configured public gates,
bootstrap CI lower bounds are 0.000000 on table metrics, and text CER is
slightly worse than Surya on the smoke (0.481629 vs 0.480531).
The same sparse-row normalizer now recognizes numbered list markers such as
1), 2., (a), and Devanagari-digit variants. This is still guarded by the
same single-cell continuation-row rule and should be treated as a parser
postprocessor until rerun on the public table gate.
The larger public OmniDocBench table40 run is more important. It reaches the
configured table-pair count (40 public table pages) but still fails validation:
table_cell_f1 improves in mean (0.323122 vs 0.280062) while losing more
pages than it wins (7 wins, 9 losses, 24 ties), and the bootstrap CI low is
negative (-0.019152). table_teds_structure also has a positive mean
(0.520959 vs 0.503603) but CI low is negative (-0.038381). table_teds_text
does not clear the margin (0.003609 improvement). The document CER gate is
also not met: only 40/100 required public document pages, and CER improvement
is only 0.000634 against a 0.005 margin.
The width-guarded v4 rerun improves the public table40 evidence but still does
not validate the full claim. Skipping table-grid normalization on tables wider
than 12 columns makes the table-cell F1 gate pass (0.339319 vs 0.280062,
CI low 0.001433) and the table TEDS-structure gate pass (0.545375 vs
0.503603, CI low 0.004790). The text-sensitive table gate still fails:
table_teds_text improves in mean (0.386013 vs 0.358057) but CI low is
-0.000220. The public document CER gate also still fails because the slice has
only 40/100 required public document pages, despite a mean CER improvement of
0.006096.
On 2026-06-14, the table100 public run was launched on k2 using
scripts/run_public_table100_claim.py and the v4 width-guard model card. The
first all-or-nothing launch stalled after three completed candidate rows. We
restarted with an isolated per-page runner, --resume-existing, and
--sample-timeout-seconds 300, preserving completed outputs and forcing future
stalls to become explicit error rows. The first measured timeout row was
omnidocbench-00216-page-dca64e05-1ce6-49ee-857a-9b5d05a87357. Do not cite this
run as a result until the validation artifact exists and has been inspected.
This validates a targeted table-ledger extraction claim, not full document SOTA. Full SOTA validation still needs broad document coverage, real scanned tables, merged-cell/header cases, true TEDS/TEDS-IOU, and public benchmark evidence.
Recognizer Hard-Failure Gate
The failed AWS G6 PaddleOCR recognizer produced a normalized 50-sample
hard-eval slice:
data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl
Use this slice as a regression gate for the next trained recognizer branch. It must remain held out; do not train on it.
Validation config:
configs/validation-paddle-hard50-tesseract.json
Run it after benchmarking a candidate recognizer against Tesseract:
ocrtech validate-claim \
--benchmark-report outputs/next-recognizer-hard50/benchmark/report.json \
--config configs/validation-paddle-hard50-tesseract.json \
--eval-manifest data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl \
--out outputs/validation-next-recognizer-hard50
This gate is intentionally framed as failure recovery:
- all
50hard rows must be present - the
38English synthetic line-crop failures must not regress versus Tesseract on CER or WER - the
12Nepali hard failures must improve WER over Tesseract by at least0.05
Passing this gate alone is not a SOTA claim. It is a recognizer branch admission test before rerunning the full mixed OCR eval, rendered document pack, and public benchmark gates.
The first line-level Tesseract/Paddle composite run exposed a benchmark-lifecycle
bug, not a model result: every candidate row crashed with exit_code=-11 because
PaddleOCR was initialized in the parent process and then inherited by forked
timeout workers. After changing timeout mode to initialize engines inside the
watchdog worker, the rerun completed all 100 rows:
outputs/experiment-line-script-paddle-hard50-v2/report.json
outputs/summary-line-script-paddle-hard50-v2/summary.md
outputs/validation-line-script-paddle-hard50-v2/validation.md
Validation status: not_validated.
- hard-50 CER: candidate
0.276489, Tesseract0.120106 - English line-crop CER: candidate
0.060429, Tesseract0.060429 - English line-crop WER: candidate
0.132259, Tesseract0.132259 - Nepali hard CER: candidate
0.960679, Tesseract0.309083 - Nepali hard WER: candidate
1.000000, Tesseract0.711607
Interpretation: line-level routing can preserve English by leaving English lines on the Tesseract primary path, but the trained PaddleOCR secondary recognizer is not admissible for Devanagari overrides.
Leakage Rules
Fatal leakage:
- same
sample_id - same image SHA-256
- same image path
- same image+text sample signature
Warning only:
- same text SHA-256
Text-only overlap is not fatal because short OCR labels repeat naturally across real corpora. It is still reported because repeated long labels may indicate bad dataset splitting.
External Benchmarks
Use public suites for broad claims and the internal Nepali/English pack for the actual target domain:
- OmniDocBench for general document parsing.
- olmOCR-bench for PDF-to-Markdown structure preservation.
- OCRBench / OCRBench v2 for VLM OCR coverage.
- Internal Nepali + English pages for Devanagari OCR, mixed-script pages, government forms, tables, captions, and noisy scans.
For table claims, cell exact match, table_cell_f1, and span-aware table_structure_similarity are v1 metrics. Add true TEDS/TEDS-IOU before claiming leadership on complex merged-cell tables.
Internal Eval Pack
See EVAL_PACK.md. The internal pack is necessary because public OCR/document benchmarks do not sufficiently cover Nepali, mixed Devanagari-English pages, and local document layouts.
Dataset Splitting
See DATASETS.md. Do not manually split data when making benchmark claims; use split-manifest so duplicate sample signatures cannot cross train/eval boundaries.