ocr-projects-archive / ocr-tech /docs /EVAL_HARNESS_GOAL.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
32.1 kB

Evaluation Harness Goal

Updated: 2026-06-20

Current Status

This is a parked claim-safety design, not the active product work. The active 2026-06-23 objective is Limbu book OCR: real book pages in, editable Unicode book text out, with line structure preserved and unresolved text surfaced for human review. Do not add generic eval-harness features from this document unless they directly unblock that OCR loop. For the current Limbu work, the useful evaluation surface is deliberately thin: frozen real-book page acceptance checks, artifact audits, page/line counts, missed/extra/blank-line rates, review-packet coverage, and reviewed-only training-pack export validation.

Goal

The goal is a claim-grade OCR evaluation harness for the Nepal/diaspora recognizer suite that makes weak publication claims difficult by default, not merely inconvenient: each language/script result must be scored only from a frozen, versioned eval pack whose source images, labels, manifest rows, source-document/page/bbox provenance, split role, domain, ground-truth trust evidence, claim eligibility, canonical pack-content digest, and script/grapheme coverage are auditable; each run must record the exact model, checkpoint, config, dictionary, prediction file, normalization profile, command, timestamp, machine, pack-content digest, and artifact hashes; each metric must be recomputed from canonical predictions as corpus micro edit-distance CER over both Unicode codepoints and UAX #29 extended grapheme clusters, with exact-line accuracy, matched/unmatched rows, worst examples, confidence intervals, and paired comparison tests when claims compare systems; recognizer-only oracle-crop scores must remain separate from page-level OCR scores that include detection, crop lineage, reading order, and alignment; and the harness must fail or hard-warn on missing images, duplicate IDs, low match rate, train/dev/test leakage, checkpoint selection on test data, synthetic or transliterated data presented as real evidence, machine-converted labels presented as human-verified evidence, human GT without review/agreement evidence, eval packs too small for the claimed conclusion, uncovered script inventories, and aggregate averages that hide per-language, per-script, per-domain, or per-GT-status failures. A reviewer opening one output directory should be able to decide what was scored, against which ground truth, how trustworthy that ground truth is, whether the test set was frozen and unseen for selection, what normalization and metric were used, how uncertain the number is, what the failures look like, what script material was not covered, and whether the result is safe for a paper claim.

Requirements

  • Eval packs are versioned evidence objects. Each pack must include images/, a label file, a manifest JSONL, and a summary/datasheet containing source documents, page/bbox provenance, language, script, domain, split role, GT source, GT trust level, known leakage status, claim eligibility, and a pack_content_sha256 over the canonical row identity/text/provenance fields so pack drift is detectable across normalization, scoring, audit, and comparison.
  • Metrics are canonical and micro-averaged. The harness must compute total-edit-distance / total-reference-length CER, not per-line macro averages. It must report both codepoint CER and grapheme-cluster CER, plus exact-line accuracy, matched/unmatched counts, prediction match rate, total reference units, and worst-line examples.
  • Point estimates are not enough for claims. Every claim-facing metric must include confidence intervals computed by resampling the scoring units, and same-pack system comparisons must include a paired test over per-row errors. The report must show the effective sample size per language/script/domain and hard-warn when a pack is too small for the claimed conclusion.
  • PaddleOCR training metrics are not claim metrics. PaddleOCR RecMetric reports exact-line acc and norm_edit_dis as one minus the average per-sample normalized Levenshtein distance; this may be useful during training, but it is not the corpus-level micro CER used for claims.
  • Recognition and page OCR are separate tasks. Oracle-crop recognizer results are valid recognizer evidence only. Page-level OCR claims require page images, detector output, crop/reading-order/alignment logic, and a separate end-to-end score.
  • Unicode handling is explicit. Text is normalized with a named profile, defaulting to NFC with whitespace preserved. The audit must reject a stale or unsupported profile that no longer matches the metric recomputation path. Metrics must also report how many raw GT/reference and prediction strings changed under that profile, and the audit must recompute those counts from canonical row artifacts so hidden normalization effects cannot disappear into a single CER. Grapheme metrics use Unicode extended grapheme clusters; any script-specific tailoring must be recorded.
  • Human verification is first-class metadata. Machine-converted GT may be scored, but the output must label it as machine-converted. Paper-grade human-reading claims require either fully reviewed GT or a documented native-reader calibration sample. Human-verified GT must record who reviewed it, how much was double-reviewed, and the observed disagreement or correction rate; a trust label without review evidence is not claim evidence.
  • Script coverage is evidence, not decoration. Each eval pack must report the codepoint, grapheme-cluster, digit, punctuation, conjunct/combining-mark, and language-specific inventory coverage it exercises, plus gaps against the intended script inventory. A low CER on a pack that omits common graphemes or conjuncts must not be eligible for broad script support claims.
  • Reports are inspectable. Each run should emit eval-run.json, metrics.json, metrics.md, predictions.tsv, source-audited unmatched.tsv, worst-lines.csv, worst-pages.csv, sample-review.csv, sample-review.html, confusions.json, command.txt, and a human-readable claim-safety.md report. Worst-example rows must carry the row identity, hashes, language/script/domain/GT/split, claim eligibility, source-document/page/bbox provenance, and reference-artifact trust fields needed to inspect a failure without reverse-engineering the rest of the run.
  • Reporting is per language/script/domain. Shared recognizers, such as the Devanagari model, must still report Majhi, Magar, Sunuwar-Devanagari, Magahi, Bhojpuri, Thami, Wambule, and Darai separately. Native-script targets such as Limbu, Sunuwar/Jenticha, Kirat Rai, Lepcha, Tirhuta, Gurung Khema, and Newa must stay separate from fallback or synthetic-only results.

Current Implementation Slice

  • tools/run_recognizer_eval.py scores an existing recognizer prediction file against a PaddleOCR-style label file. It records non-empty model identity plus config/checkpoint/dictionary, SHA-256 identities for local checkpoint/config/dictionary artifacts and ties those identities back to the visible run fields, prediction, label, manifest, summary/datasheet, script inventory, train/dev manifest, command, machine platform/Python metadata, normalization, eval-pack version, frozen-pack declaration, selection declaration, per-row crop image SHA-256 identities and source-document/page/bbox provenance in predictions.tsv and worst-lines.csv, deterministic row- bootstrap 95% confidence intervals for codepoint CER, grapheme CER, and exact-line accuracy, observed and missing required script codepoint/grapheme inventory coverage, and claim-safety warnings, and emits a direct claim-safety.md report in the score directory. A claim-safe recognizer run now requires those artifacts to be readable local files whose hashes can be audited and at least 30 scored rows per emitted language/script/domain/GT/split slice so tiny or pooled smoke packs cannot pass as paper evidence. Declared manifest crop-image hashes must match actual crop bytes; missing script inventory files or missing required codepoints/graphemes are claim-safety blockers, and rows declared human_verified must carry human review evidence such as reviewer or review-artifact metadata.
  • tools/audit_recognizer_eval_pack.py audits a recognizer oracle-crop eval pack before scoring. It verifies label/image/manifest/summary integrity, duplicate labels, manifest identifiers, and manifest image paths/basenames, manifest text/hash consistency, source-document/ page/bbox coverage, language/script/domain/GT/split metadata, claim eligibility, frozen-pack declaration, unseen-for-selection declaration, and minimum scoring-unit counts overall and per language/script/domain/GT/split slice.
  • tools/normalize_recognizer_eval_pack.py converts existing PaddleOCR-style recognizer packs into a self-contained canonical pack with copied images, normalized eval.txt, manifests/recognizer-eval.jsonl, and summary.json. It preserves legacy provenance fields, fills only explicitly provided defaults, hashes copied images, and fails on missing images, duplicate labels, duplicate manifest IDs or image paths/basenames, or manifest/label text mismatches.
  • configs/recognizer-eval-pack-normalization-v1.json records reproducible normalization recipes for the current real recognizer packs: Newa, Sunuwar, Lepcha, Limbu, Kirat Rai, and shared Devanagari. tools/run_recognizer_pack_normalization.py validates those recipes, supports dry-runs, materializes selected packs through the normalizer, writes per-pack audit/recognizer-eval-pack-audit.{json,md} artifacts, and records pack audit results in normalization-run.json.
  • configs/eval-target-coverage-v1.json records the intended language/script coverage sets: the broader first-ten document OCR targets and the current real recognizer packs. tools/validate_eval_target_coverage.py compares those targets against the normalization and eval-run recipe registries so missing packs, missing eval recipes, non-claim-ready targets, and shared-pack language/script mismatches that could hide per-language failures are explicit gaps. Target eval_status values are constrained to known states, and only registered_real_pack can become coverage-ready; partial, shared, missing, or invented status labels cannot silently pass as claim support.
  • tools/run_page_ocr_eval.py scores saved page-level OCR document.json outputs against structured references. It emits an explicit eval_mode=page_level_ocr artifact, separates page OCR from recognizer oracle-crop metrics, records model/run provenance with local checkpoint/config/dictionary SHA-256 identities, records scored source image and document.json SHA-256 identities plus the embedded document.source_path in page-metrics.csv, rejects claim-safe runs whose document.source_path does not match the eval-manifest input, records and audits the page pack pack_content_sha256, recomputes page text micro CER over codepoints and grapheme clusters, audits page-metrics.csv row identity back to the eval manifest, resolved document.json, scorer metadata defaults, row claim eligibility, and parsed reference fields, reports auditable per-language/script/domain/GT/split page slices recomputed from page-metrics.csv, existing document metrics such as reading-order pair accuracy, table scores, figure scores, reconstruction metrics, page text exact accuracy, canonical page prediction/reference text pairs, per-row reference path/hash/trust artifacts, source-document/page/bbox provenance columns, reference coverage, missing document/reference rows, row-level source-document/page/bbox/ claim-eligibility and language/script/domain/GT metadata coverage, row-level claim eligibility, line-bbox IoU matching diagnostics when references provide explicit reading-order bboxes, worst-page review CSV/HTML artifacts, deterministic page-bootstrap 95% confidence intervals for page text CER, observed and missing required script codepoint/grapheme inventory coverage from reference text, and train/dev/eval leakage checks with recorded train-manifest hashes. Page GT declared as machine_converted_spot_checked requires an eligible native-reader calibration artifact, matching the recognizer-side trust gate. Page claim-safe runs require at least 30 scored pages per emitted language/script/domain/GT/split slice, and page score directories also emit claim-safety.md so they can be reviewed without a recipe wrapper. Duplicate page manifest sample IDs or image paths and stale manifest image hashes fail before scoring, and missing script inventory files or missing required codepoints/graphemes are claim-safety blockers. Page rows declared human_verified must also carry human review evidence rather than relying on a trust label alone.
  • tools/run_eval_output_command.py materializes canonical eval outputs through a recorded external command before scoring. In recognizer mode it writes one prediction file from a label pack; in page mode it runs once per eval-manifest row and requires each sample to produce document.json. It records the exact expanded command(s), command template, invocation, timestamp, stdout/stderr log paths, output hashes/sizes, per-page input image paths/hashes, model/checkpoint/config/dictionary metadata, machine details, and failure details with machine/invocation and label or eval-manifest hashes when a command does not produce the required artifact. When --model-config is provided without --command, it uses the repo's candidate engine defaults: recognizer mode calls tools/predict_recognizer_eval.py, while page mode calls ocrtech parse-doc.
  • tools/predict_recognizer_eval.py turns a candidate model card into a canonical recognizer prediction file for oracle-crop eval packs. It reads the label file, runs the audited candidate engine on each crop, writes image<TAB>prediction<TAB> confidence, and records model-card, label, output, machine, and command hashes.
  • tools/prepare_native_spotcheck_bundle.py creates native-reader review bundles from recognizer labels, and tools/apply_native_spotcheck_reviews.py ingests completed review CSVs into calibration artifacts. The calibration step validates statuses, emits reviewed-label and manifest-override artifacts, measures machine-converted label CER against corrected/native-reviewed text, requires reviewer identity on reviewed rows, and records whether the spot-check is eligible evidence for treating a pack as machine_converted_spot_checked. Scorers bind that calibration evidence to the current run's language, script, domain, machine_converted source GT status, and machine_converted_spot_checked recommendation. Bundle summaries record source file hashes, machine metadata, timestamp, and review artifact paths.
  • tools/build_recognizer_leakage_manifest.py turns recognizer train/dev label files into JSONL leakage manifests with sample IDs, image hashes, text hashes, and combined sample hashes. These manifests are the standard input for scorer-side train/dev/eval leakage checks; text-only mode exists for corpora where image files are intentionally unavailable. Page-derived crop manifests must also preserve source_sample_id, and the page scorer/audit treat train source_sample_id overlap with eval page sample_id as hard leakage.
  • docs/EVAL_HARNESS_RUNBOOK.md gives the checked operational recipe for chaining normalization, leakage-manifest generation, output materialization, recognizer scoring, native-reader calibration, and page-level scoring. It documents the exact claim-safety flags and the distinction between repo candidate model-card runs and external PaddleOCR/HF command templates.
  • configs/eval-harness-recipes-v1.json records concrete current recipe instances for Newa, Sunuwar, Lepcha, Limbu, Kirat Rai, shared Devanagari, and packaged HF/Paddle recognizer smoke paths. tools/validate_eval_harness_recipes.py verifies that referenced local paths exist, runs the canonical recognizer eval-pack audit for each path-complete recognizer pack, statically audits page eval manifests for source-image hashes, page provenance, row claim eligibility, human-review evidence, and claim-ready references plus minimum overall and per-slice scoring-unit counts, and separates structural recipe validity from claim readiness blockers, including missing, malformed, or recipe-contradicting summary_file metadata for language/script/task/dataset/domain/GT/split/version/eligibility, missing or malformed script_inventory_file, incomplete row provenance, pack-audit warnings or issues, missing, malformed, ineligible, or mismatched recipe-level native spot-check calibration when default or audited row GT is machine_converted_spot_checked, scorer-supported normalization profile declarations, external command templates missing required input/output placeholders or containing unsupported placeholders, unresolved external prediction command templates, commands still marked as requiring operator resolution, unhashable checkpoint/config/dictionary artifacts, malformed or empty leakage_sources, declared checkpoint selection on the test pack, and declared known leakage.
  • tools/run_eval_recipe.py is the recipe-level recognizer orchestrator. It validates the selected recipe, builds or wires train/dev leakage manifests, materializes predictions through the recorded runner, scores the canonical predictions, and writes recipe.json, recipe-validation.json, recipe-run.json, root-level audit.json, root-level claim-safety.md, materialization logs, leakage artifacts, and the scorer output under one reviewable run directory. The root claim-safety report carries the score-mode-specific recognizer or page facts from the recipe, score, and audit JSON files, including script-inventory coverage, metadata contradictions, uncertainty intervals, leakage status, and the score-mode review artifact list, so it can be audited for stale reviewer-facing summaries, root recipe.json is tied back to the selected recipe in the hashed registry file, root recipe-validation.json is tied back to recipe-run.json and recomputed from the current validator, recipe-run timestamp/machine/command provenance is checked, root materialization/audit/claim-safety output pointers must point at the run directory's own artifacts, root audit summaries must preserve current score-audit issue lists, warnings, claim-safe status, and score-dir identity, root score summary paths and hashes must match score/metrics.json and score/eval-run.json, root train-manifest path/hash identities must match the score eval-run leakage inputs, the hashed recipe registry file must sit under the declared recipe root, and the root prediction/materialization paths, command template, output report, and materialized model/checkpoint/config/dictionary metadata are tied back to the scored prediction file hash and score eval-run metadata. Failed recipe attempts write recipe-run-error.json with timestamp, machine, command, recipe validation, registry hash, root, output directory, exception type, message, and traceback so a failed run is still attributable.
  • tools/compare_recognizer_eval_runs.py compares two completed recognizer oracle-crop score directories on the same frozen eval pack. It pairs canonical predictions.tsv rows by sample_id, verifies eval-pack/run identity including pack_content_sha256 and row GT identity, emits comparison.json, comparison.md, paired-rows.tsv, and command.txt, reports candidate and baseline micro CER over codepoints and graphemes, paired wins/losses/ties over row-level CER, deterministic paired-row bootstrap confidence intervals for the CER improvement, and an exact two-sided sign-test p-value. Its audit mode checks run provenance, recomputes the comparison, and rejects stale JSON, Markdown, command, or paired-row artifacts.
  • tools/compare_page_ocr_eval_runs.py does the same for completed page-level OCR score directories, using page-eval-run.json, page-metrics.csv, and text-pairs.tsv. It verifies page eval-pack identity and page/reference text row identity including pack_content_sha256, emits comparison.json, comparison.md, paired-pages.tsv, and command.txt, reports paired page text codepoint/grapheme CER improvements, deterministic paired-page bootstrap intervals, and an exact two-sided sign-test p-value, and can audit command/provenance plus the comparison artifacts back to the current score directories.
  • tools/audit_eval_run.py audits completed recognizer or page score directories, or a recipe-run root with score/, for required artifacts, run metadata, metric arithmetic, timestamp and command-file consistency, machine platform/Python provenance, recorded input-file hash consistency, scorer-supported normalization profile consistency, normalization-change summary recomputation from recognizer predictions.tsv or page text-pairs.tsv, model-artifact hash and run-field consistency, recognizer predictions.tsv source-row recomputation from the recorded label/prediction/manifest files, row-level edit-distance and per-language/script/domain/GT/split slice recomputation, recognizer crop-image and manifest-declared image-hash verification, recognizer source label/prediction recomputation for label-line, matched-line, unmatched-label, unmatched-prediction, match-rate metrics, deterministic bootstrap uncertainty intervals, and script inventory recomputation from recognizer GT rows, recognizer unmatched.tsv identity/hash/score recomputation from the source label and prediction files, recognizer confusions.json recomputation from canonical predictions, page reference-artifacts.tsv recomputation from scored reference files, page text-pairs.tsv recomputation from scored document/reference text, page confusions.json recomputation from those canonical text pairs, page text CER row and total recomputation from canonical page text pairs, page row provenance recomputation from source manifests, scored source-image/ document.json hash verification, page bootstrap uncertainty interval recomputation, script inventory recomputation from page reference text, and page-metric identity recomputation from manifest/document/reference sources, run-level language/script/domain/GT/split list recomputation from canonical row artifacts, page line-alignment.csv row-content and detection/alignment recomputation from scored document/reference files, including predicted/reference line IDs, bboxes, text, IoU, and per-line text CER, page recognizer normalizer/defaulted-metadata count checks against the recorded summary/datasheet, direct-run summary/datasheet metadata consistency checks against run language, script, domain, GT, split, eval-pack version, frozen/unseen, default claim-eligibility, and task fields, dictionary coverage recomputation from recognizer GT rows or page reference text, with GT/reference codepoints outside the recorded dictionary treated as a claim-safety blocker, missing.tsv recomputation from the eval manifest, document outputs, and reference files, slice totals, duplicate scored sample/image IDs in completed output artifacts, ambiguous recognizer manifest image keys, duplicate page manifest image paths, page manifest image-hash mismatches, leakage-check presence and leakage-overlap recomputation from train/eval sources, native spot-check calibration summary checks against the recorded calibration artifact, metadata-contradiction recomputation from source provenance signals, provenance/metadata coverage recomputation from source manifests, page reference-coverage and manifest/scored/missing/claim-eligible count recomputation from source artifacts, durable claim-safety blockers such as metadata contradictions, missing manifests or summary/datasheets, unrun leakage checks, leakage overlaps, low match rate, unsafe domains, untrusted GT, missing provenance, missing uncertainty evidence, too-small claim-facing sample size, missing script inventory evidence, missing required script codepoints/graphemes, zero-length recognizer GT rows or page text references, recognizer row-level claim ineligibility, and incomplete page coverage, warning/claim-safety consistency, metrics.md consistency against metrics.json, recomputed worst-example CSVs, reviewer CSV order/fields, and reviewer HTML contents, direct score-directory claim reports whose core lines are checked against JSON run facts, recipe-root claim reports whose recognizer/page score facts are checked against the nested score directory, and materialization artifacts whose timestamp, machine platform/Python and invocation metadata, JSON records, model/checkpoint/config/dictionary metadata, static recipe-level dictionary coverage checks against recognizer GT or page reference codepoints, per-record commands/log paths, command-template file, Markdown report, recognizer prediction path/hash or page document-output directory, and output hashes are checked against the score eval-run facts plus claim-report artifacts whose top-level report is also checked against recipe and score JSON facts.
  • The scorer computes corpus micro CER over NFC codepoints and Unicode extended grapheme clusters, exact-line accuracy, matched/unmatched rows, per-language/script/ domain/GT/split slice metrics, worst lines/pages, review CSV/HTML, and recognizer/page confusion counts.
  • Claim-safety warnings now cover missing manifest or summary, missing eval-pack version, missing script inventory files, missing required script codepoints or graphemes, GT/reference codepoints outside the recorded dictionary, unhashable model artifacts, zero-length recognizer GT rows or page text references, non-frozen or selection-used eval packs, low prediction match rate, mixed languages/scripts/domains/GT statuses/splits, unsafe domains, untrusted GT, metadata contradictions such as synthetic/transliterated provenance declared as real material or machine-converted provenance declared as human_verified, missing source-document/page/bbox/claim-eligibility provenance, explicit row-level claim ineligibility, missing page row source-document/page/bbox/ claim-eligibility/language/script/domain/GT metadata, normalizer-filled metadata defaults from canonical pack summaries, missing human-review evidence for human_verified rows, missing, mismatched, or ineligible native spot-check calibration when machine_converted_spot_checked is claimed, known leakage, unrun leakage checks, and train/dev/eval overlaps from provided JSONL manifests.
  • A Newa recognizer artifact was produced at outputs/evals/newa-prachalit/newa-native-rescore-2026-06-19-zenodo/. It is intentionally not claim-safe: GT is machine-converted, the pack is not declared unseen for selection, page provenance and claim eligibility are missing, and no train/dev leakage manifest was provided.
  • The same Newa eval pack has a standalone audit at outputs/evals/newa-prachalit/newa-real-eval-pack-audit-2026-06-19/. Basic pack integrity passes (3180/3180 labels matched to manifest rows and images), but claim readiness fails because the pack is not declared unseen for selection, has no page metadata, lacks row-level claim eligibility, and uses machine-converted GT.

Remaining Gaps

  • Resolve remaining remote-only checkpoint/package locations in configs/eval-harness-recipes-v1.json and replace placeholder external commands with runnable per-family commands. The recipe registry exists and validates local paths, and tools/run_eval_recipe.py can execute resolved recognizer recipes, but several language-specific PaddleOCR checkpoints are not packaged locally or hash-verifiable, so the validator must keep them out of claim-ready status.
  • Extend page-level evaluation with live detector output capture and crop lineage from model runners; the current page harness scores saved structured documents and emits per-line IoU/alignment rows when references provide explicit line bboxes, but it does not yet run the detector/recognizer pipeline itself.
  • Archive and review the materialized canonical eval packs under outputs/eval-packs/. The six current real recognizer packs now exist with per-pack audits, but all six are structurally non-claim-ready because of machine-converted GT, missing or defaulted provenance/claim metadata, and for some packs selection-status blockers.
  • Fill the target coverage gaps reported by tools/validate_eval_target_coverage.py. The current coverage config validates, but is intentionally not coverage-ready: English, Tamang, Magar, Wambule, Bantawa/Rai, Kulung, Yakkha, and Tharu/Rana Tharu still lack registered real eval packs, and the current registered real-pack recipes remain non-claim-ready because model commands, GT trust, provenance, calibration, or row eligibility blockers remain unresolved. A target also remains non-coverage-ready unless its eval_status is exactly registered_real_pack; use partial_real_recognizer_pack, shared_devanagari_pack, or missing_real_eval_pack only when the resulting gap is intended and visible.
  • Generate and wire train/dev leakage manifests for each actual recognizer family and pass them to scoring runs. The manifest builder exists, but each family still needs its concrete train/dev label paths recorded in the run recipes, including page-origin source_sample_id for crops selected from full-page eval material.
  • Apply native-reader spot-check/calibration to the actual language eval packs, with enough reviewed rows per script/domain to justify any machine_converted_spot_checked upgrade. The harness now has artifacts and scorer wiring for this, but the reviewed human evidence still has to be collected.

Research Basis