ocr-projects-archive / ocr-tech /docs /EVAL_HARNESS_RUNBOOK.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
44.4 kB

Eval Harness Runbook

Updated: 2026-06-20

This is the operational path for producing a claim-grade OCR eval directory. The goal is not to get a number quickly; it is to make the number hard to misread. Each run should leave enough artifacts for a reviewer to answer what was scored, which model produced it, which ground truth was used, whether the pack was frozen and unseen for selection, what leakage checks ran, and why the result is or is not safe for a paper claim.

Recipe-First Eval

Use this path when a registered recipe exists for either recognizer_oracle_crop or page_level_ocr. Recipes keep validation, leakage checks, output materialization, scoring, root reports, and audit metadata under one output directory.

Registry Validation

When a recipe exists in configs/eval-harness-recipes-v1.json, use it as the default entry point. This keeps validation, leakage checks, prediction materialization, and scoring under one output directory.

Before scoring legacy source packs, validate or run the normalization registry:

uv run python tools/run_recognizer_pack_normalization.py \
  --config configs/recognizer-eval-pack-normalization-v1.json \
  --root . \
  --all \
  --out outputs/eval-packs/normalization-runs/<run-id> \
  --dry-run \
  --require-valid

Drop --dry-run only when you intentionally want to materialize canonical packs under outputs/eval-packs/<pack-version>/. A materialized pack gets its own audit/recognizer-eval-pack-audit.json and .md; read those before wiring the pack into a claim run.

Then validate intended language/script coverage against the normalization and eval-run recipe registries:

uv run python tools/validate_eval_target_coverage.py \
  --coverage-config configs/eval-target-coverage-v1.json \
  --normalization-config configs/recognizer-eval-pack-normalization-v1.json \
  --eval-recipes configs/eval-harness-recipes-v1.json \
  --root . \
  --out outputs/evals/target-coverage.json \
  --require-valid

Do not add --require-coverage-ready until every target in the selected claim scope has a normalized eval pack, runnable eval recipe, and no claim blockers. If a target points at a shared aggregate pack, the validator must show matching target language/script metadata or an explicit gap; do not use a shared pack average as evidence for an individual language claim. Coverage target status is also gated: eval_status must be one of the known states, and only registered_real_pack is allowed to become coverage-ready. Use partial_real_recognizer_pack, shared_devanagari_pack, or missing_real_eval_pack when a target is deliberately not ready for claims. For book OCR targets, set requires_page_eval_recipe and requires_book_candidate_evidence on the target set or target. That makes coverage depend on a page-level recipe with book-candidate evidence from the scoring/admission lane, not just an oracle-crop recognizer score or a loose page replay. The validator also reports whether the linked book candidate is operationally admitted and claim-safe, so non-claim-safe Sirijonga and Devanagari book evidence show up as separate gaps.

uv run python tools/validate_eval_harness_recipes.py \
  --recipes configs/eval-harness-recipes-v1.json \
  --root . \
  --require-valid

Each recipe eval_pack must declare normalization_profile: NFC_preserve_whitespace, matching the scorer/auditor recomputation contract, and a readable, parseable summary_file and script_inventory_file. The recipe validator checks the summary JSON object and inventory JSON shape before a recipe is considered structurally valid, and a summary without dataset/version/eval-pack metadata blocks claim readiness. Summary/datasheet declarations for language, script, dataset, task, domain, GT status, split, pack version, frozen/unseen status, and default claim eligibility must agree with the recipe metadata; contradictions block claim readiness. Do not paper over a pack that needs another normalization policy; add scorer support and audit coverage first, then change the recipe. The script inventory is a JSON object with required_codepoints and required_graphemes lists. Codepoints may be written as U+XXXX values or literal single characters; graphemes are literal strings. The scorer records observed coverage from GT/reference text, and claim-safe runs fail if any required item is missing. When a runner dictionary is a readable local file, the recipe validator also compares its non-whitespace codepoints to recognizer GT or page reference text; out-of-vocabulary GT/reference codepoints block claim readiness before scoring. If the recipe default or audited row metadata uses machine_converted_spot_checked GT, the recipe must also declare a readable JSON eval_pack.spotcheck_calibration_file; the recipe validator checks its eligibility/recommendation/source-status fields and language/script/domain consistency, and recipe execution passes that artifact into the scorer, where it is hashed and audited.

Pack Content Digest Contract

Every claim-facing eval pack summary/datasheet must include pack_content_sha256. This is not a cosmetic version string. It is the immutable content identity that lets scoring, audit, and comparison prove that the score still belongs to the same source pack.

For recognizer oracle-crop packs, the digest binds each canonical row's sample ID, source image hash, NFC-normalized GT text hash, language, script, domain, GT status, split, and claim eligibility. For page-level OCR packs, the digest binds each canonical page row's sample ID, input image hash, reference artifact hash, reference text hash, reference fields, reference trust metadata, source-document/page/bbox provenance, language, script, domain, GT status, split, and claim eligibility.

Claim-safe scoring must fail when the summary digest is missing or stale. Completed score directories must record the recomputed value in metrics.json and in the mode-specific run file (eval-run.json for recognizer, page-eval-run.json for page OCR). Audit must recompute the digest from current source artifacts and reject any mismatch in the run JSON, metrics JSON, or summary/datasheet. Same-task comparison tools must treat the digest as eval-pack identity, so a candidate and baseline cannot be compared as claim-safe if their pack_content_sha256 values differ.

When a test or fixture intentionally edits pack content before scoring, regenerate the summary digest before running the scorer. When content changes after scoring, do not refresh the digest; that is tampering or drift, and audit should fail.

For page_level_ocr recipes, eval_pack.eval_manifest is the canonical page manifest and materialization writes per-page document.json files under the root documents/ directory. External page command templates must write either {document_json} directly or write document.json below {output_dir}. The recipe validator statically audits page manifests for readable source images and hashes, source-document/page/bbox provenance, row claim eligibility, human-review evidence for human_verified rows, parseable references, claim-ready reference metadata, and minimum scoring-unit counts overall and per language/script/domain/GT/split slice; any page audit issue or warning blocks claim readiness. For path-complete recognizer packs, the recipe validator also runs tools/audit_recognizer_eval_pack.py; any pack-audit issue or warning becomes a claim-readiness blocker, including overall and language/script/domain/GT/split sample-size warnings below the claim minimum. leakage_sources must also be parseable and non-empty; page-level leakage manifests must satisfy the page manifest schema, and recognizer label leakage sources must contain valid rows with existing images. A recipe is not claim-ready merely because its label, manifest, summary, script inventory, and image paths exist.

For Limbu, keep the pack counts straight. The canonical materialized recognizer pack outputs/eval-packs/limbu-gorkhapatra-real-v1/ has 1,829 labels and 1,829 manifest rows, not 39. The number 39 belongs to an older operational page run's unresolved review queue and must not be described as the available Limbu eval set. The current Limbu recognizer pack is structurally auditable but not claim-ready: it is machine-converted, has 0 claim-eligible rows, is not declared unseen for selection, and is missing source-document/page/bbox provenance. The separate CDC Maths PDF text-layer pack has 81 silver rendered-line rows, split 69 train / 12 eval; use it for recognizer bootstrap/training experiments, not for claim-safe evaluation. For page_level_ocr recipes with ocr_output_kind=live_detector, the recipe must also declare runner.output_audit bound to {output_dir} so page scoring is paired with an operational artifact audit. Limbu page recipes must run audit-limbu-output and record strict flags including --require-capture-prep, prepared-image quality thresholds (--min-capture-prepared-width, --min-capture-prepared-height, --min-capture-prepared-entropy, --min-capture-prepared-luminance-stddev, and --min-capture-prepared-edge-stddev), --require-no-pending-review, --require-no-dropped-image-lines, output quality/volume gates (--min-line-count, --min-average-line-confidence, and --min-quality-score), and the script-specific --require-script value for their target, for example limbu_sirijonga for Sirijonga-script Limbu or devanagari_limbu for Devanagari-written Limbu. This keeps page scoring tied to the operational artifact audit instead of allowing a claim-ready registry entry for an OCR bundle that would fail provenance, capture-quality, output-volume, output-quality, review-resolution, or complete-line-retention gates.

Then run the selected recipe:

uv run python tools/run_eval_recipe.py \
  --recipes configs/eval-harness-recipes-v1.json \
  --recipe-id <recipe-id> \
  --root . \
  --out outputs/evals/<run-id> \
  --timeout-seconds 3600 \
  --min-match-rate 0.99

Add --require-claim-safe only when the recipe validator reports no claim blockers and the scorer is expected to emit no warnings. If the recipe uses an external command template, replace the placeholder command in the recipe first; unresolved templates are rejected by default. Recognizer external templates must include {label_file}, {data_dir}, and {prediction_file}; page external templates must include {input_path} plus {document_json} or {output_dir}. Unknown placeholders are validation errors, so command templates must use only the mode-supported placeholders documented by tools/run_eval_output_command.py. Recipes that declare runner.selected_checkpoint_on_test or non-empty known_leakage remain structurally valid but are blocked from claim readiness because the scorer will warn on those same facts at runtime. Runner checkpoint, config, and dictionary values must point to readable local files, not directories, because claim-safe scores record and audit their SHA-256 hashes. Recognizer and page eval-pack summaries must also carry pack_content_sha256, the canonical digest over sample ID, input/reference hashes, normalized GT text or structured-reference text, provenance, language, script, domain, GT status, split, and claim eligibility. The scorers recompute this digest from the rows they actually scored; claim-safe runs fail when the summary digest is missing or stale.

The recipe runner writes the root-level files a reviewer should open first: recipe-run.json, audit.json, and claim-safety.md. For page-level recipes with runner.output_audit, execution runs that audit once per materialized page output directory before scoring and records output-audit/output-audit-run.json plus output-audit/command-template.txt and Markdown with commands, logs, return codes, and available audit artifact hashes. The runner honors runner.output_audit.result_filename when recording the per-page audit JSON; if it is omitted, the default is limbu-output-audit.json. The root audit requires each recorded per-page audit JSON to still exist with the recorded SHA-256. It also requires the output-audit command template sidecar when recipe-run.json records an output-audit command, rejects a missing or stale sidecar, and replays each per-page output-audit command from that template using the recorded output directory. It also replays the recorded stdout/stderr log SHA-256 values for each per-page audit command. It requires output-audit/output-audit-run.md and checks that the Markdown report still reflects the recorded count, per-page output audit rows, log hashes, and recorded audit artifact hashes. The report summarizes what was scored, the GT trust metadata, frozen/unseen declarations, leakage status, micro-CER metrics, output-audit status, warnings, and review artifact locations. The root report is audited against recipe-run.json, the score eval-run JSON, and metrics.json, with mode-specific expectations for recognizer oracle-crop and page-level OCR score facts, so stale top-level summaries fail audit. The recipe-root audit also checks that materialize/output-command-run.json matches the materialization summary embedded in recipe-run.json, that materialize/command-template.txt and materialize/output-command-run.md match the recorded command/output summary, that materialization timestamp, machine platform/ Python metadata, invocation, per-record command, and stdout/stderr log paths and SHA-256 values are present and replay against current log bytes. It also replays materialization record commands from the recorded command template using the recorded recognizer or page materialization fields; if those fields are missing, the root audit fails instead of skipping command replay. For recognizer roots, the root predictions.txt path and hash must be the same prediction artifact recorded by the score eval-run JSON. For page roots, each materialization record must carry the source page image path/hash, the recorded hash is rechecked against current bytes, outputs.documents_dir must match the score eval-run documents_dir, and materialized document.json outputs must live under that directory. Recipe-root materialization model/checkpoint/config/dictionary metadata is also checked against the score eval-run metadata. If materialization fails before a score directory exists, keep materialize/output-command-run-error.json: it carries the timestamp, machine, invocation, model/checkpoint/config/dictionary metadata, command template, and label or eval-manifest path/hash used for the failed attempt. If the recipe wrapper fails, keep recipe-run-error.json: it records the timestamp, machine, command, recipe registry hash, selected recipe validation, root, output directory, exception type, message, and traceback. The underlying recognizer or page score directory also writes its own claim-safety.md, so direct scorer runs are reviewable without a recipe wrapper.

Audit the finished run directory before reporting the number:

uv run python tools/audit_eval_run.py \
  --run-dir outputs/evals/<run-id> \
  --out outputs/evals/<run-id>/audit.json \
  --require-audit-pass

The audit recomputes recognizer row edits and aggregate micro-CER from score/predictions.tsv; a metric file that is internally ratio-consistent but does not match the emitted GT/prediction rows must fail audit. The run machine metadata must include concrete platform and python values, and normalization_profile must match the scorer-supported NFC_preserve_whitespace profile used by the audit recomputation path. The audit also recomputes normalization_summary from predictions.tsv for recognizer runs and text-pairs.tsv for page runs; those fields count raw GT/reference and prediction strings changed by NFC, so normalization-heavy scores remain visible instead of being hidden behind normalized CER. The audit also reconstructs predictions.tsv row identity, GT text, prediction text, prediction score, crop hash, language/script/domain/GT/split, and row claim-eligibility fields from the recorded label, prediction, and manifest sources. The same file carries language/script/domain/GT/split, claim-eligibility, and crop image SHA-256 values, so recognizer slice metrics, claim-eligibility provenance, and crop-image identity are recomputed from the emitted row artifact, not trusted as separate aggregates. worst-lines.csv is also audited against those canonical prediction rows, including crop hashes, trust metadata, claim eligibility, and source-document/page/bbox provenance, so a failure sample cannot be made more claim-friendly by editing the review artifact after scoring. Recognizer label-line count, scored-line count, matched-line count, unmatched label count, unmatched prediction count, and prediction match rate are recomputed from the recorded source label and prediction files. Recognizer normalizer_counts in metrics.json and eval-run.json are also checked against the recorded summary/datasheet counts, so defaulted pack metadata cannot be hidden by editing both score JSON files after scoring. Recognizer and page summary/datasheet files are semantically audited too: if a hash-valid summary declares language, script, domain, GT status, split, eval-pack version, frozen/unseen status, default claim eligibility, or task metadata that contradicts the scored run metadata, the audit fails. Run-level language/script/domain/GT/split lists in eval-run.json are also checked against the unique values in predictions.tsv. predictions.tsv also carries source-document/page/bbox provenance columns, and the audit reconstructs those values from the recorded label/manifest sources. For page runs, the audit also recomputes line-alignment.csv row contents from the scored document.json and reference JSON, including predicted/reference line IDs, line bboxes, text, IoU, and per-line text CER, then recomputes line detection counts, precision/recall/F1, mean IoU, and aggregate metric means from line-alignment.csv and page-metrics.csv. The audit also reconstructs page-metrics.csv row identity from the eval manifest, resolved document.json, embedded document.source_path, scorer metadata defaults, and parsed reference fields, so language/script/domain/GT/split, source-document/page/bbox provenance, claim_eligible, and reference_fields cannot be changed after scoring without failing audit. Run-level language/script/domain/GT/split lists in page-eval-run.json must match the unique values in page-metrics.csv. reference-artifacts.tsv carries the exact reference path, SHA-256, scoreable reference fields, text hash, and reference trust metadata for each scored page, and the audit recomputes it from the eval manifest and reference files. worst-pages.csv is audited against page-metrics.csv plus reference-artifacts.tsv, including input/document hashes, language/script/domain/GT metadata, claim eligibility, source-document/page/bbox provenance, reference hashes, and reference trust fields. page-metrics.csv carries raw page text edit/ref counts and exact text-match flags, text-pairs.tsv carries the exact prediction/reference page text used for those counts, and the audit recomputes the text-pair artifact from scored document.json text plus reference text before rolling it into page text micro-CER, text exact-accuracy totals, and per-language/script/domain/GT/split page slices. It also records source page image and document.json SHA-256 values, which the audit rechecks so a page score cannot point at mutated OCR outputs after the fact. Train/dev/eval leakage reports are recomputed from the recorded train manifests and canonical eval rows/manifests, including checked state, row counts, and overlap sets. For page-derived crop training or checkpoint-selection data, include source_sample_id on train rows; scorer and audit treat overlap between train source_sample_id and eval page sample_id as hard leakage even when crop image hashes and line text hashes do not match the full page row directly. Native spot-check calibration summaries embedded in run and metric JSON are checked against the recorded calibration artifact. Metadata contradictions are recomputed from manifest/reference provenance signals and canonical row domain/GT fields. Recognizer provenance coverage and page row metadata coverage counts are recomputed from their source manifests, and page reference coverage is recomputed from page-metrics.csv. Page manifest/scored/missing/claim-eligible counts are recomputed from the eval manifest, document outputs, references, and claim-eligibility metadata. Recognizer and page score directories also carry deterministic bootstrap uncertainty summaries in metrics.json; the audit recomputes those intervals from predictions.tsv or page-metrics.csv. Claim-safe runs currently require at least 30 scored rows/pages, so one-page smoke checks and tiny oracle-crop packs must remain non-claim-safe even when their point CER is perfect. Recognizer and page score directories also carry script inventory summaries in metrics.json and run JSON; the audit recomputes observed codepoints/graphemes from recognizer GT rows or page reference text and verifies the inventory file hash. A missing inventory file, a stale inventory summary, or missing required codepoints/graphemes remains a claim-safety blocker even when CER is low. They also carry dictionary_coverage in metrics.json; the audit rehashes the recorded dictionary file, recomputes the non-whitespace GT/reference codepoints, and blocks claim-safe status when the ground truth contains codepoints outside the dictionary vocabulary. metrics.md and claim-safety.md must surface the same dictionary coverage, normalization-change, and zero-length GT/reference counters, and the audit treats stale Markdown as an output-directory integrity failure. Recognizer metrics record zero_length_gt_lines, and page metrics record zero_length_text_references; the audit recomputes both from canonical rows and blocks claim-safe status when any scored GT/reference text is empty. The audit also derives claim-safety blockers from durable run facts, so a tampered claim_safe=true field without matching warnings still fails if the artifacts show metadata contradictions, a missing manifest or summary/datasheet, an unrun leakage check, leakage overlaps, low recognizer match rate, unsafe GT/domain status, missing provenance, missing uncertainty evidence, missing script inventory evidence, missing required script inventory items, zero-length GT/reference text, too-small claim-facing sample size, recognizer row-level claim ineligibility, or incomplete page coverage. The direct and recipe-root claim-safety.md reports are also checked against their JSON run and metric facts, using recognizer or page score fields according to eval_mode, including score timestamp, machine, command provenance, hashed checkpoint/config/dictionary artifact identity, normalization-change, dictionary-coverage, script-inventory gaps, metadata contradictions, uncertainty intervals, zero-length GT/reference counters, and per-language/script/domain/GT/split slice metrics, so stale or aggregate-only reviewer-facing summaries fail audit instead of merely having the right section headings. Direct and recipe-root score reports must also list the review artifacts a reviewer needs to open, including command.txt, canonical row artifacts, worst-example CSVs, sample-review CSV/HTML files, confusions, and page alignment/reference artifacts. For recipe-root audits, recipe.json must still match the selected recipe in the hashed source registry and recipe-validation.json must still match recipe-run.json; the embedded validation is also recomputed from the current recipe registry and eval-pack audit. Recipe-run timestamp, machine, command, and root output pointers are audited too. Root audit.json and the embedded recipe-run audit summary must preserve current score-audit issues, warnings, claim-safe status, and score-dir identity, and the recipe-run score summary must point to and hash the current score/metrics.json plus the mode-specific score run file (score/eval-run.json for recognizer roots, score/page-eval-run.json for page roots). The root train-manifest path/hash list must also match the score eval-run leakage inputs, and the hashed recipe registry must be under the declared recipe root. Do not hand-edit those files after scoring. For page-level Limbu book-candidate recipes that use tools/score_limbu_book_ocr_candidate.py, the recipe validator also audits the declared evidence chain. It reads the score wrapper, admission report, metrics, and review-coverage report; verifies recorded SHA-256 values; requires champion metrics; requires review-coverage counts for missed and extra lines; and blocks candidates with champion regressions or uncovered review regions. It also parses the recipe command_template and requires the scorer command to bind the same --book-output, --book-audit, --champion-metrics, and --review-coverage-report paths recorded in the evidence chain. A command string or free-text claim blocker is not enough evidence for current-best registration, and stale evidence cannot be attached to a weaker replay command. The score-level metrics.md report is checked against metrics.json for core counts, CER values, slices, reference coverage, metric means, and warnings. Recognizer unmatched.tsv row identity, label image hashes, prediction scores, and source-level match/count metrics are recomputed from the recorded source label and prediction files, recognizer confusions.json is recomputed from canonical predictions, page confusions.json is recomputed from canonical text-pairs.tsv prediction/reference text, and page missing.tsv is recomputed from the eval manifest, document outputs, and reference files. Worst-example CSVs, reviewer CSV order and fields, and reviewer HTML samples are also checked against the canonical scored rows, so stale or hand-edited failure samples do not satisfy reviewability. Completed score artifacts are checked for duplicate scored sample IDs, image paths, document paths, and missing-row keys; duplicate IDs are not only a pre-normalization pack failure.

For paper-claim numbers, also add --require-claim-safe.

Recognizer Oracle-Crop Eval

Use this manual path for line-crop recognizers only. Do not use it for page-level OCR claims.

Manual Path

  1. Normalize the eval pack into the canonical schema.
uv run python tools/normalize_recognizer_eval_pack.py \
  --label-file data/<pack>/eval.txt \
  --data-dir data/<pack> \
  --manifest-file data/<pack>/manifest.jsonl \
  --summary-file data/<pack>/summary.json \
  --out outputs/eval-packs/<language>-<pack-version> \
  --dataset <dataset-id> \
  --language <language> \
  --script <script> \
  --domain real_print \
  --gt-status machine_converted \
  --split eval \
  --eval-pack-version <pack-version> \
  --frozen-eval-pack \
  --unseen-for-selection \
  --no-claim-eligible

If the normalizer reports missing source document, page, bbox, GT status, or claim eligibility, do not hide it with defaults. Fix the pack or keep the result non-claim-safe. If a recognizer manifest row declares an image sha256, the scorer verifies that it matches the current crop bytes. A stale declared hash fails scoring and fails later audit if an older score directory points at a mutated manifest.

  1. Build train/dev leakage manifests for every model family used to select or train the checkpoint.
uv run python tools/build_recognizer_leakage_manifest.py \
  --label data/<train-pack>/train.txt::data/<train-pack>::train \
  --label data/<train-pack>/eval.txt::data/<train-pack>::dev \
  --out outputs/leakage/<model-family>-train-dev.jsonl \
  --summary outputs/leakage/<model-family>-train-dev.summary.json \
  --source <model-family> \
  --language <language> \
  --script <script> \
  --domain real_print \
  --gt-status machine_converted

Use --text-only only when the corpus intentionally has no images. A text-only manifest can catch text leakage, but it cannot catch duplicate crop images. If crops were cut from a page eval candidate, the leakage manifest must preserve the origin page as source_sample_id; otherwise a line-crop pack can look clean under exact image/text hash checks while still being selected from the eval page.

  1. Materialize predictions through a recorded command.

For repo candidate model cards, omit --command and provide --model-config:

uv run python tools/run_eval_output_command.py \
  --mode recognizer \
  --model-config outputs/<model-package>/model-card.json \
  --out outputs/evals/<run-id>/materialize \
  --system-name <system-name> \
  --model-name <model-name> \
  --checkpoint <local-checkpoint-file> \
  --config <local-training-or-inference-config-file> \
  --dictionary <local-dictionary-file> \
  --label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
  --data-dir outputs/eval-packs/<language>-<pack-version> \
  --prediction-file outputs/evals/<run-id>/predictions.txt \
  --timeout-seconds 3600

For external PaddleOCR/HF runners, pass --command explicitly. The command must write {prediction_file} in image<TAB>prediction<TAB>confidence format and may use only {label_file}, {data_dir}, {prediction_file}, and {out_dir}.

uv run python tools/run_eval_output_command.py \
  --mode recognizer \
  --command 'bash scripts/predict_external_recognizer.sh --labels {label_file} --data {data_dir} --out {prediction_file}' \
  --out outputs/evals/<run-id>/materialize \
  --system-name <system-name> \
  --model-name <model-name> \
  --checkpoint <local-checkpoint-file> \
  --config <local-training-or-inference-config-file> \
  --dictionary <local-dictionary-file> \
  --label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
  --data-dir outputs/eval-packs/<language>-<pack-version> \
  --prediction-file outputs/evals/<run-id>/predictions.txt \
  --timeout-seconds 3600
  1. Score the predictions.
uv run python tools/run_recognizer_eval.py \
  --label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
  --data-dir outputs/eval-packs/<language>-<pack-version> \
  --prediction-file outputs/evals/<run-id>/predictions.txt \
  --out outputs/evals/<run-id>/score \
  --manifest-file outputs/eval-packs/<language>-<pack-version>/manifests/recognizer-eval.jsonl \
  --summary-file outputs/eval-packs/<language>-<pack-version>/summary.json \
  --script-inventory-file outputs/eval-packs/<language>-<pack-version>/script-inventory.json \
  --train-manifest outputs/leakage/<model-family>-train-dev.jsonl \
  --model-name <model-name> \
  --checkpoint <local-checkpoint-file> \
  --config <local-training-or-inference-config-file> \
  --dictionary <local-dictionary-file> \
  --language <language> \
  --script <script> \
  --domain real_print \
  --gt-status machine_converted \
  --split eval \
  --eval-pack-version <pack-version> \
  --frozen-eval-pack \
  --unseen-for-selection \
  --no-selected-checkpoint-on-test \
  --no-claim-eligible \
  --min-match-rate 0.99

Only add --require-claim-safe when every warning is expected to be absent. If the GT status is machine_converted_spot_checked, also pass --spotcheck-calibration-file. Recipe runs can declare the same artifact under eval_pack.spotcheck_calibration_file so the calibration evidence is part of the versioned recipe instead of an invisible operator-only flag. For a claim-safe recognizer run, --checkpoint, --config, and --dictionary must point to readable local files. The scorer records their SHA-256 hashes in eval-run.json, and tools/audit_eval_run.py re-verifies those hashes and checks that the visible run fields still match the hashed artifact identities. The scorer also records whether all non-whitespace GT codepoints are present in the dictionary; missing GT codepoints are a claim-safety warning and are independently recomputed during audit. The same audit checks hash-valid summary/datasheet metadata against the direct run's recorded language/script/domain/GT/split, version, frozen/unseen, default claim-eligibility, and task metadata. --script-inventory-file must also point to a readable local JSON file; it is hashed in eval-run.json, summarized in metrics.json, and audited against the emitted GT rows.

After scoring, audit the score directory:

uv run python tools/audit_eval_run.py \
  --run-dir outputs/evals/<run-id>/score \
  --out outputs/evals/<run-id>/score/audit.json \
  --require-audit-pass

Same-Pack Recognizer Comparisons

When a paper claim compares two recognizers, do not compare separate aggregate numbers by eye. Score both systems on the same eval pack, audit both score directories, then run the paired comparison tool:

uv run python tools/compare_recognizer_eval_runs.py \
  --candidate-score-dir outputs/evals/<candidate-run>/score \
  --baseline-score-dir outputs/evals/<baseline-run>/score \
  --candidate-name <candidate-name> \
  --baseline-name <baseline-name> \
  --out outputs/evals/<comparison-run> \
  --min-pairs 30 \
  --require-claim-safe

The tool pairs rows by sample_id, checks that eval-pack identity including pack_content_sha256 and row GT identity match, and emits comparison.json, comparison.md, paired-rows.tsv, and command.txt. It reports codepoint and grapheme micro-CER deltas, paired wins/losses/ties, deterministic paired-row bootstrap confidence intervals for the improvement, and a two-sided exact sign-test p-value. A comparison is claim-safe only when both input score directories are claim-safe, the rows are fully paired on the same eval pack, the bootstrap lower bound is above zero, the sign test passes the configured alpha, the candidate actually improves the baseline, and comparison timestamp/machine/ command provenance is present and internally consistent.

Audit the comparison directory before using it in a claim:

uv run python tools/compare_recognizer_eval_runs.py \
  --audit-only \
  --comparison-dir outputs/evals/<comparison-run> \
  --require-claim-safe

For page-level OCR system comparisons, use the page comparison tool instead of the recognizer comparison tool:

uv run python tools/compare_page_ocr_eval_runs.py \
  --candidate-score-dir outputs/evals/<candidate-page-run> \
  --baseline-score-dir outputs/evals/<baseline-page-run> \
  --candidate-name <candidate-name> \
  --baseline-name <baseline-name> \
  --out outputs/evals/<page-comparison-run> \
  --min-pairs 30 \
  --require-claim-safe

It pairs pages by sample_id, checks page eval-pack identity including pack_content_sha256 plus page/reference text identity, and emits comparison.json, comparison.md, paired-pages.tsv, and command.txt for page text codepoint/grapheme CER deltas, paired wins/losses/ties, paired-page bootstrap intervals, and a two-sided exact sign-test p-value. Audit it the same way:

uv run python tools/compare_page_ocr_eval_runs.py \
  --audit-only \
  --comparison-dir outputs/evals/<page-comparison-run> \
  --require-claim-safe

Native-Reader Spot Check

Machine-converted labels may be scored, but they are not human-verified evidence. Use this path to create auditable calibration evidence.

uv run python tools/prepare_native_spotcheck_bundle.py \
  --label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
  --data-dir outputs/eval-packs/<language>-<pack-version> \
  --manifest-file outputs/eval-packs/<language>-<pack-version>/manifests/recognizer-eval.jsonl \
  --prediction-file outputs/evals/<run-id>/predictions.txt \
  --out outputs/spotchecks/<language>-<pack-version> \
  --sample-size 50 \
  --seed 20260619 \
  --title '<language> native-reader spot check'

After a reviewer exports the completed CSV:

uv run python tools/apply_native_spotcheck_reviews.py \
  --review-file outputs/spotchecks/<language>-<pack-version>/reviewed-spotcheck.csv \
  --out outputs/spotchecks/<language>-<pack-version>/calibration \
  --language <language> \
  --script <script> \
  --domain real_print \
  --source-gt-status machine_converted \
  --min-usable-rows 30 \
  --max-machine-grapheme-cer 0.02 \
  --max-unusable-rate 0.05

If the calibration output says claim_evidence_eligible=false, do not mark the pack machine_converted_spot_checked. Completed rows without reviewer identity are not eligible calibration evidence; fix the review export rather than overriding the gate. The scorer also requires the calibration artifact's language, script, domain, source_gt_status=machine_converted, and recommended_pack_gt_status=machine_converted_spot_checked to match the scored run.

Page-Level OCR Eval

Use this path for end-to-end page claims. Recognizer-only scores cannot support a page OCR claim because detection, crop lineage, and reading order are absent.

  1. Materialize page outputs through a recorded command.
uv run python tools/run_eval_output_command.py \
  --mode page \
  --model-config outputs/<page-model-package>/model-card.json \
  --out outputs/page-evals/<run-id>/materialize \
  --system-name <system-name> \
  --model-name <model-name> \
  --checkpoint <local-checkpoint-file> \
  --config <local-training-or-inference-config-file> \
  --dictionary <local-dictionary-file> \
  --eval-manifest data/<page-eval-pack>/manifests/eval.jsonl \
  --documents-dir outputs/page-evals/<run-id>/documents \
  --timeout-seconds 3600

For an external runner, pass --command and write {document_json} for each input page. Page commands may use only {input_path}, {sample_id}, {output_dir}, {document_json}, {documents_dir}, and {out_dir}.

  1. Score saved document.json outputs.
uv run python tools/run_page_ocr_eval.py \
  --eval-manifest data/<page-eval-pack>/manifests/eval.jsonl \
  --documents-dir outputs/page-evals/<run-id>/documents \
  --ocr-output-kind live_detector \
  --out outputs/page-evals/<run-id>/score \
  --summary-file data/<page-eval-pack>/summary.json \
  --script-inventory-file data/<page-eval-pack>/script-inventory.json \
  --model-name <model-name> \
  --checkpoint <local-checkpoint-file> \
  --config <local-training-or-inference-config-file> \
  --dictionary <local-dictionary-file> \
  --eval-pack-version <pack-version> \
  --frozen-eval-pack \
  --unseen-for-selection \
  --no-selected-checkpoint-on-test \
  --train-manifest outputs/leakage/<model-family>-train-dev.jsonl

For direct live-detector page scoring, claim-safe mode requires a passing output audit for every scored page output directory. By default the scorer looks for limbu-output-audit.json with passed=true beside each document.json; use --require-output-audit, --output-audit-filename, and --output-audit-passed-field when scoring a different page-output audit format. The page auditor replays those per-page audit artifacts from page-metrics.csv and fails if a recorded page_output_audit summary no longer matches the current files, so deleting or editing an output audit after scoring invalidates the score directory.

A paper-safe page claim requires reference text plus page/line layout references that let the scorer report detection/alignment/reading-order behavior. If line bboxes are absent, the result is not a full page-level OCR claim. Each page manifest row must also carry explicit language, script, domain, and gt_status metadata plus source-document, page, bbox, and claim-eligibility provenance; CLI defaults are recorded as fallbacks, not as row-level claim evidence. The page summary must carry a pack_content_sha256 over the canonical page row, input-image, reference-artifact, reference-text, and provenance identity; claim-safe scoring rejects summaries whose digest is missing or stale. Page manifest sample_id and image_path values must be unique, and any declared row sha256 must match the actual input image bytes; duplicate IDs, duplicate input images, or stale image hashes fail before scoring. If page GT status is machine_converted_spot_checked, pass --spotcheck-calibration-file; without an eligible calibration artifact the page result is not claim-safe. For a claim-safe page run, --checkpoint, --config, and --dictionary must point to readable local files whose SHA-256 hashes can be recorded and audited. --script-inventory-file must point to a readable local JSON file; the scorer records which required codepoints/graphemes were covered by reference text and the auditor recomputes that summary from the eval manifest and references. --ocr-output-kind must identify what kind of document outputs were scored. Use live_detector for normal end-to-end page OCR outputs, oracle_line_boxes for diagnostic runs where reference/PDF line boxes are fed to recognizers, and sidecar_fixture only for tests or synthetic fixtures. For Limbu, the detector diagnostic requires scoring the same frozen page pack once as live_detector and once as oracle_line_boxes, then comparing the two score directories with tools/compare_page_ocr_eval_runs.py.

Audit the page score directory:

uv run python tools/audit_eval_run.py \
  --run-dir outputs/page-evals/<run-id>/score \
  --out outputs/page-evals/<run-id>/score/audit.json \
  --require-audit-pass

Claim-Safe Gates

A recognizer or page result is not paper-safe unless all of these are true:

  • Eval pack is frozen, versioned, and declared unseen for model/checkpoint selection.
  • The score records a manifest and summary/datasheet for the eval pack.
  • The score records a script inventory file, its hash, observed codepoint/grapheme coverage, and no missing required codepoints or graphemes.
  • Manifest row IDs and input image paths are unique, recognizer manifest image paths/basenames cannot collide with basename-based prediction matching, and declared source-image hashes match the current files; stale recognizer crop hashes, duplicate page manifest IDs or image paths, stale page manifest image hashes, and duplicate completed score artifact IDs fail audit.
  • Every scored row has source-document, page, bbox, language, script, domain, split, GT status, and claim eligibility metadata, and every scored row is claim eligible.
  • GT status is human_verified or backed by eligible native-reader calibration as machine_converted_spot_checked.
  • Rows declared human_verified carry human review evidence such as reviewed_by, review_artifact, double-review, agreement, or correction-rate metadata; the trust label alone is not claim evidence.
  • The domain is real target material, not synthetic or transliterated evidence.
  • Row/reference provenance does not contradict the declared domain or GT status: synthetic/transliterated provenance cannot be claimed as real material, and machine-converted provenance cannot be claimed as human_verified.
  • Train/dev leakage manifests were passed and checked.
  • Model artifacts are hash-verifiable: recognizer and page runs require checkpoint, config, and dictionary files, and the run fields must match the hashed artifact identity entries.
  • Model identity is recorded: recognizer and page runs require a non-empty model name, and page runs require a non-empty system name.
  • No hard leakage exists by sample ID, image hash, image path, or combined sample hash.
  • The prediction match rate is at least the configured threshold.
  • No scored recognizer GT row or page text reference is zero-length.
  • Headline reporting is per language/script/domain/GT status, not a pooled average hiding weak languages, and every emitted language/script/domain/GT/split slice has at least the claim-facing minimum scoring units.
  • Same-pack recognizer comparisons use tools/compare_recognizer_eval_runs.py and pass its paired bootstrap/sign-test gates; separate aggregate score directories are not enough evidence for a comparative claim.
  • The score directory contains claim-safety.md with What Was Scored, Ground Truth And Pack, Metrics, and Warnings sections.