prosperious's picture
Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified
|
Raw History Blame Contribute Delete
31.8 kB

Validation Design

The claim target is not "we ran OCR and it looked good." The claim target is:

ocrtech is better than named baselines on held-out Nepali + English OCR and document-structure extraction, under measured gates.

Until the validation report says validated, do not claim SOTA.

Claim-grade validation.json files record provenance for the benchmark report, validation config, eval manifest, train manifests, candidate model card, and fallback model card used by validate-claim. review-claim cross-checks those hashes against experiment.json, so a validation artifact from a different benchmark, eval split, validation config, model card, fallback card, or training-manifest set cannot be reused as portfolio evidence.

The SOTA review profile is paired with these claim-grade validation configs:

  • configs/validation-internal-sota-v1.json
  • configs/validation-public-omnidocbench-sota-v1.json
  • configs/validation-public-real5-sota-v1.json

Each one covers the required_validation_gates declared in configs/claim-review-sota-v1.json. Use the slot-specific config when running run-experiment or validate-claim; otherwise portfolio review may have metrics and baselines but still lack the passed gates required for the paper claim.

The built-in fallback validation config is intentionally broader than a plain OCR smoke check. If validate-claim is called without --config, it now gates Nepali CER/WER, English non-regression, table cell and TEDS-text quality, reading order, document reconstruction, and figure detection/caption quality. That default remains smaller than the SOTA configs above, but it should not allow a "document OCR" claim to pass on text-only evidence.

Evidence Required

A defensible run needs all of this:

  • A claim-ready image eval pack from ocrtech create-eval-pack, plus public benchmark runs.
  • A candidate model card from ocrtech create-model-card and a passing ocrtech audit-model report.
  • Training data prepared with audit-manifest and split-manifest.
  • Held-out eval manifest that was never used for recognizer, corrector, threshold, or prompt tuning.
  • Image/PDF eval inputs normalized to OCR-supported file suffixes before any claim-grade PaddleOCR/Tesseract/Surya/VLM comparison.
  • For routed or thresholded candidates, a separate calibration experiment artifact that records how the threshold was chosen.
  • Train manifests for leakage checks.
  • Paired benchmark outputs for every baseline on the same input files.
  • Per-sample metrics, not only aggregate averages.
  • For glm-ocr and paddleocr-vl, a real runnable command hook that emits Markdown or document.md, not a placeholder env var.
  • Slice coverage for Nepali, English, multi-page documents, tables, reading order, figures/charts, forms, scans, photos, skew, blur, compression, and low contrast.
  • Gates that name the baseline, metric, slice, direction, required margin, and confidence requirement.
  • Paired win/loss evidence, a minimum paired win-rate requirement when the claim needs broad per-document improvement, and a nonparametric significance check when the claim requires it.
  • Failure-case report, not just the winning average.

Public Benchmark Gate

Generated eval packs are development evidence. They are not enough for SOTA wording. Before a paper claim can say "state of the art," at least one public document parsing benchmark must be converted into an auditable manifest and run as a normal ocrtech run-experiment.

Known public benchmark registry:

ocrtech list-public-benchmarks

Current registry entries:

  • opendatalab/OmniDocBench: real-world document parsing benchmark, dataset page https://huggingface.co/datasets/opendatalab/OmniDocBench
  • allenai/olmOCR-bench: PDF-to-Markdown benchmark with unit-test style assertions, dataset page https://huggingface.co/datasets/allenai/olmOCR-bench
  • PaddlePaddle/Real5-OmniDocBench: real-world physical reconstruction benchmark with scanning, warping, screen-photo, illumination, and skew scenarios, dataset page https://huggingface.co/datasets/PaddlePaddle/Real5-OmniDocBench

The first implemented public converter is:

ocrtech prepare-public-benchmark opendatalab/OmniDocBench \
  --out data/public/omnidocbench \
  --limit 100 \
  --language english \
  --require-tables

The converter was written only after checking the actual dataset structure: OmniDocBench.json contains page_info, page_info.page_attribute, and layout_dets; image files resolve as images/{page_info.image_path}; table ground truth is stored in table-block html.

Every public benchmark manifest row must include:

  • metadata.public_benchmark = true
  • metadata.benchmark_name
  • metadata.source_url
  • metadata.reference_path, unless the benchmark has a separate official scorer integration
  • metadata.slices including public plus benchmark-specific slices
  • an existing image_path
  • a matching image sha256, when the manifest records one
  • a matching metadata.text_sha256, when the manifest records one

Audit it before running:

ocrtech audit-public-benchmark data/public/omnidocbench/manifests/omnidocbench-eval.jsonl \
  --benchmark opendatalab/OmniDocBench \
  --out outputs/public-omnidocbench-audit

This audit intentionally fails if provenance, references, images, or recorded hashes are missing or stale. Do not use --allow-missing-reference-paths for claim runs unless an official external scorer is wired and documented in the experiment report.

Portfolio review for SOTA claims should use:

ocrtech review-claim \
  --experiment outputs/internal-experiment/experiment.json \
  --experiment outputs/public-omnidocbench/experiment.json \
  --config configs/claim-review-sota-v1.json \
  --model-config outputs/figure-caption-table-lexicon-v1/model-card.json \
  --out outputs/claim-review-sota-v1

The config requires internal Nepali-English evidence plus public benchmark evidence. It should fail until those public experiments actually exist.

Claim-grade validation configs can also set min_pair_win_rate on a gate. This is intentionally different from the reported win_rate: win_rate is computed over non-tied pairs, while min_pair_win_rate is checked as wins divided by all paired samples. That distinction matters for document OCR claims. A result with one win and many ties can have win_rate = 1.0, but it has low pair coverage and should not satisfy a broad "better than baseline" claim unless the gate is explicitly a no-regression/tie-acceptable gate.

The public OmniDocBench-vs-Surya validation gate is:

ocrtech validate-claim \
  --benchmark-report outputs/experiment-public-omnidocbench-smoke-v1/benchmark/report.json \
  --config configs/validation-public-omnidocbench-surya.json \
  --eval-manifest data/public/omnidocbench-smoke/manifests/omnidocbench-eval.jsonl \
  --candidate-model-config runs/recognizer/package/model-card.json \
  --fallback-model-config runs/vlm-fallback/package/model-card.json \
  --out outputs/validation-public-omnidocbench-smoke-v1

It requires at least 100 public document-parsing samples and at least 40 table samples, then gates candidate-vs-Surya CER, table cell F1, structure-only TEDS, and text-sensitive TEDS. A 5-page smoke run is allowed as engineering evidence, but it must remain not_validated.

The first audited manifest sized for this gate is:

data/public/omnidocbench-table100/manifests/omnidocbench-eval.jsonl

Its audit artifact is outputs/public-omnidocbench-table100-audit/public-benchmark-audit.json; the audit passed with 100 public/document/table English samples. The corresponding long benchmark run is tracked under outputs/experiment-public-omnidocbench-table100-width-guard-v4/. Treat that run as incomplete until benchmark/report.json, summary/summary.json, validation/validation.json, and table-cell-analysis/table-cell-analysis.json exist.

The completed width-guard-v4 Table100 run is an instructive non-passing result:

  • rows: 200/200, with one shared 300s timeout for candidate and Surya
  • paired scored samples: 99
  • candidate improves public table metrics over Surya: table_cell_f1 0.484364 vs 0.406694, table_teds_structure 0.720160 vs 0.687550, table_teds_text 0.565284 vs 0.543987
  • candidate does not improve OCR CER: 0.613201 vs Surya 0.612623
  • validation status: not_validated

For long-run status files, distinguish package readiness from claim readiness. evidence_package_ready means the expected rows and final report artifacts are present. paper_claim_ready must only be true when the validation report itself passes. A completed benchmark package with validation.claim_status = not_validated is useful evidence, but not a publishable SOTA claim.

Long public runs should use benchmark checkpointing and per-page timeouts:

ocrtech benchmark ... \
  --inputs-from-manifest data/eval/manifest-with-refs.jsonl \
  --resume-existing \
  --sample-timeout-seconds 300

--inputs-from-manifest loads authoritative image paths from the manifest and defaults --eval-manifest to the same file when it is not supplied. This avoids shell-generated input lists drifting away from the manifest that contains reference paths and slice labels. --resume-existing scores existing document.json outputs and only runs OCR for missing pages. --sample-timeout-seconds runs each OCR call behind a hard process watchdog on platforms that support fork; a stalled backend is terminated and recorded as a page-level TimeoutError, then the benchmark continues. Platforms without fork fall back to the Python signal timer. A timeout is still a candidate/baseline failure for validation; it is not a pass-through. For timeout runs, the benchmark now constructs the OCR engine inside the watchdog child process for each page. This avoids inheriting already-initialized PaddleOCR/PaddlePaddle C++ runtime state across fork, which produced exit_code=-11 worker crashes in the first line-level Paddle composite run.

Table scoring matches each reference table to the best unused predicted table before averaging table metrics. This matters on public document pages where an engine may emit extra table fragments before the main table. Positional table-index matching is too brittle for claim evidence.

When the evaluator changes, do not rerun slow OCR just to update metrics. Use:

ocrtech rescore-benchmark \
  --benchmark-report outputs/experiment-public-omnidocbench-smoke-v1/benchmark/report.json \
  --eval-manifest data/public/omnidocbench-smoke/manifests/omnidocbench-eval.jsonl \
  --out outputs/experiment-public-omnidocbench-smoke-v2-rescored/benchmark

The rescore command copies the existing document.json, Markdown, table, and figure outputs into the new benchmark directory so downstream diagnostics such as analyze-table-cells remain usable.

When evaluating table postprocessors, do not rely only on separate candidate and baseline OCR runs. Independent OCR/layout calls can introduce run-to-run variance that makes a small table postprocessor look better or worse than it is. Use a fixed-source derived comparison to isolate the postprocessor:

ocrtech derive-table-postprocess-benchmark \
  --benchmark-report outputs/experiment-public-omnidocbench-table40-normalize-rows-v3/benchmark/report.json \
  --source-baseline surya \
  --candidate-baseline candidate \
  --eval-manifest data/public/omnidocbench-table40/manifests/omnidocbench-eval.jsonl \
  --out outputs/experiment-public-omnidocbench-table40-derived-postprocess-v1/benchmark

This command copies the chosen source baseline outputs, applies the current reference-free table postprocessors to the copied candidate document.json, and rescored both systems against the same references. The result is valid evidence for the postprocessor contribution only. It is not an end-to-end OCR model card and cannot support a broad OCR SOTA claim by itself.

The current normalized public candidate is ocrtech-surya-tesseract-figure-table-lexicon-normalize-v2. Its table-grid normalization is runtime-only and reference-free: empty columns, separator-only columns, and exact duplicate adjacent columns are removed. On the 5-page OmniDocBench smoke it improves table metrics over Surya, but the run remains not_validated because the public sample count, table sample count, CER margin, and bootstrap confidence requirements are not met.

Reference Format

Text-only reference:

page.png.ref.txt

Structured reference:

{
  "text": "नेपाल सरकार",
  "reading_order": ["p0-l0", "p0-l1", "p0-l2"],
  "tables": [
    [
      ["नाम", "उमेर", "जिल्ला"],
      ["राम", "३२", "काठमाडौं"]
    ]
  ],
  "metadata": {
    "source": "hand_labeled"
  }
}

For manually reviewed Gorkhapatra language pages, reading_order may also use structured objects. The scorer uses each object's id for ordering and keeps the text as reviewer-visible reference context:

{
  "reading_order": [
    {"id": "p0-l0", "type": "heading", "text": "मगर भाषा पृष्ठ"},
    {"id": "p0-l1", "type": "paragraph", "text": "मुख्य समाचार"}
  ]
}

Complex table references can also use cell objects with spans:

{
  "tables": [
    {
      "cells": [
        {"row": 0, "col": 0, "text": "नाम", "rowspan": 1, "colspan": 2},
        {"row": 1, "col": 0, "text": "राम"},
        {"row": 1, "col": 1, "text": "१२"}
      ]
    }
  ]
}

The nested-list grid form remains supported. Span-aware references make table_exact_match and table_structure_similarity sensitive to missing rowspans/colspans even when flattened table text is correct.

The candidate parser can apply a conservative repeated-header span postprocessor: adjacent identical cells in the first header rows are collapsed into a single colspan cell only when the following row has distinct child columns. This lets document.json and exported table HTML represent the simple merged-header pattern in complex-table-ledger. Treat this as a heuristic parser repair until it is validated on real scanned tables with varied merged cells and rowspans.

Supported structured metrics:

  • quality_score
  • cer
  • wer
  • text_cer
  • text_wer
  • document_reconstruction_cer
  • document_reconstruction_wer
  • document_reconstruction_text_similarity
  • document_reconstruction_similarity
  • figure_detection_precision
  • figure_detection_recall
  • figure_detection_f1
  • figure_caption_cer
  • reading_order_pair_accuracy
  • table_cell_precision
  • table_cell_recall
  • table_cell_f1
  • table_exact_match
  • table_structure_similarity
  • table_teds_structure
  • table_teds_text

document_reconstruction_text_similarity is 1 - CER over rendered Markdown only. document_reconstruction_similarity is stricter: when the reference also declares reading order, tables, or figures, the reconstruction score is capped by those structure metrics. This prevents a parser from receiving full document reconstruction credit by emitting Markdown-shaped text while failing to populate structured document.json tables or figures.

Validation Config

Example:

{
  "candidate": "ours",
  "min_samples": 300,
  "required_slices": {
    "nepali": 100,
    "english": 100,
    "multi_page": 25,
    "table": 50,
    "reading_order": 50,
    "figure": 30,
    "scan": 50,
    "phone_photo": 50
  },
  "bootstrap_iterations": 1000,
  "bootstrap_confidence": 0.95,
  "require_full_eval_manifest_coverage": true,
  "gates": [
    {
      "baseline": "tesseract",
      "metric": "cer",
      "direction": "lower",
      "slice": "nepali",
      "min_pairs": 50,
      "min_relative_improvement": 0.05,
      "max_p_value": 0.05
    },
    {
      "baseline": "stock-paddle",
      "metric": "cer",
      "direction": "lower",
      "slice": "english",
      "min_absolute_margin": 0.0
    },
    {
      "baseline": "glm-ocr",
      "metric": "table_cell_f1",
      "direction": "higher",
      "slice": "table",
      "min_absolute_margin": 0.01
    },
    {
      "baseline": "surya",
      "metric": "cer",
      "direction": "lower",
      "slices": ["hard_eval", "table"],
      "min_pairs": 8,
      "min_absolute_margin": 0.02
    },
    {
      "baseline": "paddleocr-vl",
      "metric": "reading_order_pair_accuracy",
      "direction": "higher",
      "slice": "reading_order",
      "min_absolute_margin": 0.01
    }
  ]
}

Use slice for a single required slice. Use slices when a gate must apply to the intersection of multiple labels, for example ["hard_eval", "table"] for table samples inside a hard-eval pocket. Do not approximate this with separate hard_eval and table gates; that allows unrelated samples to satisfy the table gate.

For claim-grade runs, set require_full_eval_manifest_coverage to true. When enabled, validate-claim fails unless the candidate and every baseline named by the configured gates have a successful row for every sample in --eval-manifest. This prevents a validation report from passing on an easy subset while silently omitting hard pages from the held-out manifest.

Run:

ocrtech preflight-claim \
  --eval-manifest data/manifests/eval.jsonl \
  --references-dir data/references \
  --baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
  --candidate-model-config runs/recognizer/package/model-card.json \
  --validation-config validation.json \
  --train-manifest data/manifests/train.jsonl \
  --out outputs/preflight

ocrtech benchmark \
  --eval-manifest data/manifests/eval.jsonl \
  --references-dir data/references \
  --baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
  --out outputs/benchmark \
  $(cut -f4 data/manifests/eval.inputs)

ocrtech validate-claim \
  --benchmark-report outputs/benchmark/report.json \
  --config validation.json \
  --eval-manifest data/manifests/eval.jsonl \
  --train-manifest data/manifests/train.jsonl \
  --out outputs/validation

preflight-claim should be the first command in a claim-grade workflow. If it fails, the benchmark run is not ready.

For explicit --validation-config runs, preflight now checks the config against the requested baselines and eval manifest before any OCR starts. It fails when the candidate label is absent from --baselines, a gate baseline is not scheduled, min_samples exceeds the manifest size, required slices are missing, or a gate's min_pairs exceeds the number of manifest rows matching the gate slice intersection. Omitting --validation-config is only a warning; it is not claim-grade evidence.

For hybrid runs with --fallback-model-config, preflight also checks that the candidate model card declares the same fallback under provenance.fallback_model_card. This catches a wrong runtime fallback flag before a benchmark produces evidence for a different composite engine than the one described by the candidate model card.

If the candidate model uses quality_select_composite, preflight-claim now checks that the model card contains calibration provenance and that the calibration eval manifest hash is different from the final claim eval manifest hash. Tuning on the same eval pack invalidates the claim path.

validate-claim exits with 0 only when the configured claim is validated. It exits with 1 when evidence is missing or a gate fails.

Each gate now reports:

  • paired sample count
  • win / loss / tie counts
  • win rate on non-tied pairs
  • bootstrap CI for mean paired improvement
  • optional one-sided sign-test p-value

For claim-grade runs, use min_pairs and max_p_value instead of relying only on a favorable mean. A narrow win on a handful of pages is not SOTA evidence.

For a single reproducible command that runs benchmark and validation together, use run-experiment; see EXPERIMENTS.md.

For the final portfolio-level claim review across the internal pack and public suites, use review-claim; see CLAIM_REVIEW.md.

For model artifact provenance, see MODELS.md. Do not use an implicit auto engine for SOTA claims.

Table Cell Analysis

Aggregate table_cell_f1 is not enough to explain whether a table-oriented candidate is actually improving table OCR. Use analyze-table-cells after a benchmark has exported per-system document.json files:

ocrtech analyze-table-cells \
  --benchmark-report outputs/experiment-table-cell-refine-empty-hard15-v1/benchmark/report.json \
  --eval-manifest data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
  --out outputs/table-cell-analysis-refine-empty-hard15-v1 \
  --candidate candidate \
  --baseline surya \
  --baseline tesseract

The command requires:

  • A benchmark report from benchmark or run-experiment.
  • Exported benchmark documents at benchmark/<system>/<input_stem>/document.json.
  • An eval manifest whose references contain structured tables.

It writes:

  • table-cell-analysis.json
  • table-cell-analysis.md

The report includes reference table count, predicted table count, duplicate overlapping table count, reference cell count, exact cell count, exact-cell rate, missing-cell count, mean cell CER, paired candidate-vs-baseline wins/losses/ties, and the highest-error candidate cells. This is the right artifact for identifying whether a table refiner is adding headroom, damaging already-correct cells, or producing structurally duplicated tables.

Interpret raw full-page Tesseract table results carefully. In the current v1 engine, Tesseract can be strong on exact image crops but still report zero structured table cells in full-page document parsing because it does not emit the table structure expected by document.json.

For the rendered hard15 table pages, reference text is the page title, while table contents live in structured tables. Do not mix these contracts: page-level cer answers whether the surrounding page text/title was recognized, and table_cell_f1 plus cell exact analysis answers whether the table content was extracted. A benchmark change that inserts table grids into Document.text without also regenerating references changes the metric definition and is invalid for this pack.

The current validation-hard15-surya.json table-cell gate can fail even with a higher mean table-cell F1 when the bootstrap confidence interval includes zero. The figure-caption/table-lexicon v3 run is the concrete example: candidate 1.000000 vs Surya 0.979167, but only 2 table pages improve and 6 tie, so the CI low is 0.000000. Treat that as evidence that the table slice is too small for a strong claim. Do not remove the CI gate after observing the result; build a larger held-out table pack.

configs/validation-table-ledger-surya.json is the first targeted table-only validation config. It requires at least 40 table samples, gates table_cell_f1 and table_exact_match against Surya with bootstrap CI above zero, and guards table_structure_similarity for non-regression. The v2 table-ledger pack (60 samples, 720 cells) passes this config:

  • table_cell_f1: candidate 1.000000, Surya 0.979167, CI low 0.012500
  • table_exact_match: candidate 1.000000, Surya 0.750000, CI low 0.150000
  • table_structure_similarity: candidate 1.000000, Surya 1.000000, non-regression guard passes

table_structure_similarity is a deterministic table-structure score over row count, column count, occupied cell coordinates, and rowspans/colspans when structured cell references are present. It is still not full HTML tree-edit TEDS. In the table-ledger v2 run it correctly shows no structure improvement over Surya because both systems already recover the simple grid structure perfectly. The candidate's validated gain is table cell-content accuracy with structure non-regression.

table_teds_structure and table_teds_text are deterministic TEDS-style ordered tree-edit scores over a table DOM made of table, tr, and td nodes. Structure mode compares row/cell tree shape plus rowspan/colspan; text mode also includes normalized cell text in td node labels. These are closer to paper-grade table metrics than the older token score, but they are still internal deterministic metrics, not a substitute for running an accepted public benchmark implementation when making broad SOTA claims.

After the span-inference postprocessor and tree metrics, the generated complex-table pack shows a structure gain over Surya (table_structure_similarity 1.000000 vs 0.875000, table_teds_structure 1.000000 vs 0.904762, table_teds_text 0.996250 vs 0.891667, table_exact_match 0.925000 vs 0.000000). This is evidence for the generated repeated-header pattern only. It does not replace a public or real scanned-table benchmark.

The public OmniDocBench smoke now also has a row-fragment normalization candidate. It improves public table metrics over Surya on the same 5-page smoke (table_cell_f1 0.564335 vs 0.296264, table_teds_structure 0.942836 vs 0.740128, table_teds_text 0.675633 vs 0.526327) and fixes one 16x5 fragmented table into the expected 3x5 shape. It is still not_validated: the sample count is far below the configured public gates, bootstrap CI lower bounds are 0.000000 on table metrics, and text CER is slightly worse than Surya on the smoke (0.481629 vs 0.480531). The same sparse-row normalizer now recognizes numbered list markers such as 1), 2., (a), and Devanagari-digit variants. This is still guarded by the same single-cell continuation-row rule and should be treated as a parser postprocessor until rerun on the public table gate.

The larger public OmniDocBench table40 run is more important. It reaches the configured table-pair count (40 public table pages) but still fails validation: table_cell_f1 improves in mean (0.323122 vs 0.280062) while losing more pages than it wins (7 wins, 9 losses, 24 ties), and the bootstrap CI low is negative (-0.019152). table_teds_structure also has a positive mean (0.520959 vs 0.503603) but CI low is negative (-0.038381). table_teds_text does not clear the margin (0.003609 improvement). The document CER gate is also not met: only 40/100 required public document pages, and CER improvement is only 0.000634 against a 0.005 margin.

The width-guarded v4 rerun improves the public table40 evidence but still does not validate the full claim. Skipping table-grid normalization on tables wider than 12 columns makes the table-cell F1 gate pass (0.339319 vs 0.280062, CI low 0.001433) and the table TEDS-structure gate pass (0.545375 vs 0.503603, CI low 0.004790). The text-sensitive table gate still fails: table_teds_text improves in mean (0.386013 vs 0.358057) but CI low is -0.000220. The public document CER gate also still fails because the slice has only 40/100 required public document pages, despite a mean CER improvement of 0.006096.

On 2026-06-14, the table100 public run was launched on k2 using scripts/run_public_table100_claim.py and the v4 width-guard model card. The first all-or-nothing launch stalled after three completed candidate rows. We restarted with an isolated per-page runner, --resume-existing, and --sample-timeout-seconds 300, preserving completed outputs and forcing future stalls to become explicit error rows. The first measured timeout row was omnidocbench-00216-page-dca64e05-1ce6-49ee-857a-9b5d05a87357. Do not cite this run as a result until the validation artifact exists and has been inspected.

This validates a targeted table-ledger extraction claim, not full document SOTA. Full SOTA validation still needs broad document coverage, real scanned tables, merged-cell/header cases, true TEDS/TEDS-IOU, and public benchmark evidence.

Recognizer Hard-Failure Gate

The failed AWS G6 PaddleOCR recognizer produced a normalized 50-sample hard-eval slice:

data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl

Use this slice as a regression gate for the next trained recognizer branch. It must remain held out; do not train on it.

Validation config:

configs/validation-paddle-hard50-tesseract.json

Run it after benchmarking a candidate recognizer against Tesseract:

ocrtech validate-claim \
  --benchmark-report outputs/next-recognizer-hard50/benchmark/report.json \
  --config configs/validation-paddle-hard50-tesseract.json \
  --eval-manifest data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl \
  --out outputs/validation-next-recognizer-hard50

This gate is intentionally framed as failure recovery:

  • all 50 hard rows must be present
  • the 38 English synthetic line-crop failures must not regress versus Tesseract on CER or WER
  • the 12 Nepali hard failures must improve WER over Tesseract by at least 0.05

Passing this gate alone is not a SOTA claim. It is a recognizer branch admission test before rerunning the full mixed OCR eval, rendered document pack, and public benchmark gates.

The first line-level Tesseract/Paddle composite run exposed a benchmark-lifecycle bug, not a model result: every candidate row crashed with exit_code=-11 because PaddleOCR was initialized in the parent process and then inherited by forked timeout workers. After changing timeout mode to initialize engines inside the watchdog worker, the rerun completed all 100 rows:

outputs/experiment-line-script-paddle-hard50-v2/report.json
outputs/summary-line-script-paddle-hard50-v2/summary.md
outputs/validation-line-script-paddle-hard50-v2/validation.md

Validation status: not_validated.

  • hard-50 CER: candidate 0.276489, Tesseract 0.120106
  • English line-crop CER: candidate 0.060429, Tesseract 0.060429
  • English line-crop WER: candidate 0.132259, Tesseract 0.132259
  • Nepali hard CER: candidate 0.960679, Tesseract 0.309083
  • Nepali hard WER: candidate 1.000000, Tesseract 0.711607

Interpretation: line-level routing can preserve English by leaving English lines on the Tesseract primary path, but the trained PaddleOCR secondary recognizer is not admissible for Devanagari overrides.

Leakage Rules

Fatal leakage:

  • same sample_id
  • same image SHA-256
  • same image path
  • same image+text sample signature

Warning only:

  • same text SHA-256

Text-only overlap is not fatal because short OCR labels repeat naturally across real corpora. It is still reported because repeated long labels may indicate bad dataset splitting.

External Benchmarks

Use public suites for broad claims and the internal Nepali/English pack for the actual target domain:

  • OmniDocBench for general document parsing.
  • olmOCR-bench for PDF-to-Markdown structure preservation.
  • OCRBench / OCRBench v2 for VLM OCR coverage.
  • Internal Nepali + English pages for Devanagari OCR, mixed-script pages, government forms, tables, captions, and noisy scans.

For table claims, cell exact match, table_cell_f1, and span-aware table_structure_similarity are v1 metrics. Add true TEDS/TEDS-IOU before claiming leadership on complex merged-cell tables.

Internal Eval Pack

See EVAL_PACK.md. The internal pack is necessary because public OCR/document benchmarks do not sufficiently cover Nepali, mixed Devanagari-English pages, and local document layouts.

Dataset Splitting

See DATASETS.md. Do not manually split data when making benchmark claims; use split-manifest so duplicate sample signatures cannot cross train/eval boundaries.