# Eval Harness Runbook Updated: 2026-06-20 This is the operational path for producing a claim-grade OCR eval directory. The goal is not to get a number quickly; it is to make the number hard to misread. Each run should leave enough artifacts for a reviewer to answer what was scored, which model produced it, which ground truth was used, whether the pack was frozen and unseen for selection, what leakage checks ran, and why the result is or is not safe for a paper claim. ## Recipe-First Eval Use this path when a registered recipe exists for either `recognizer_oracle_crop` or `page_level_ocr`. Recipes keep validation, leakage checks, output materialization, scoring, root reports, and audit metadata under one output directory. ### Registry Validation When a recipe exists in `configs/eval-harness-recipes-v1.json`, use it as the default entry point. This keeps validation, leakage checks, prediction materialization, and scoring under one output directory. Before scoring legacy source packs, validate or run the normalization registry: ```bash uv run python tools/run_recognizer_pack_normalization.py \ --config configs/recognizer-eval-pack-normalization-v1.json \ --root . \ --all \ --out outputs/eval-packs/normalization-runs/ \ --dry-run \ --require-valid ``` Drop `--dry-run` only when you intentionally want to materialize canonical packs under `outputs/eval-packs//`. A materialized pack gets its own `audit/recognizer-eval-pack-audit.json` and `.md`; read those before wiring the pack into a claim run. Then validate intended language/script coverage against the normalization and eval-run recipe registries: ```bash uv run python tools/validate_eval_target_coverage.py \ --coverage-config configs/eval-target-coverage-v1.json \ --normalization-config configs/recognizer-eval-pack-normalization-v1.json \ --eval-recipes configs/eval-harness-recipes-v1.json \ --root . \ --out outputs/evals/target-coverage.json \ --require-valid ``` Do not add `--require-coverage-ready` until every target in the selected claim scope has a normalized eval pack, runnable eval recipe, and no claim blockers. If a target points at a shared aggregate pack, the validator must show matching target language/script metadata or an explicit gap; do not use a shared pack average as evidence for an individual language claim. Coverage target status is also gated: `eval_status` must be one of the known states, and only `registered_real_pack` is allowed to become coverage-ready. Use `partial_real_recognizer_pack`, `shared_devanagari_pack`, or `missing_real_eval_pack` when a target is deliberately not ready for claims. For book OCR targets, set `requires_page_eval_recipe` and `requires_book_candidate_evidence` on the target set or target. That makes coverage depend on a page-level recipe with book-candidate evidence from the scoring/admission lane, not just an oracle-crop recognizer score or a loose page replay. The validator also reports whether the linked book candidate is operationally admitted and claim-safe, so non-claim-safe Sirijonga and Devanagari book evidence show up as separate gaps. ```bash uv run python tools/validate_eval_harness_recipes.py \ --recipes configs/eval-harness-recipes-v1.json \ --root . \ --require-valid ``` Each recipe `eval_pack` must declare `normalization_profile: NFC_preserve_whitespace`, matching the scorer/auditor recomputation contract, and a readable, parseable `summary_file` and `script_inventory_file`. The recipe validator checks the summary JSON object and inventory JSON shape before a recipe is considered structurally valid, and a summary without dataset/version/eval-pack metadata blocks claim readiness. Summary/datasheet declarations for language, script, dataset, task, domain, GT status, split, pack version, frozen/unseen status, and default claim eligibility must agree with the recipe metadata; contradictions block claim readiness. Do not paper over a pack that needs another normalization policy; add scorer support and audit coverage first, then change the recipe. The script inventory is a JSON object with `required_codepoints` and `required_graphemes` lists. Codepoints may be written as `U+XXXX` values or literal single characters; graphemes are literal strings. The scorer records observed coverage from GT/reference text, and claim-safe runs fail if any required item is missing. When a runner dictionary is a readable local file, the recipe validator also compares its non-whitespace codepoints to recognizer GT or page reference text; out-of-vocabulary GT/reference codepoints block claim readiness before scoring. If the recipe default or audited row metadata uses `machine_converted_spot_checked` GT, the recipe must also declare a readable JSON `eval_pack.spotcheck_calibration_file`; the recipe validator checks its eligibility/recommendation/source-status fields and language/script/domain consistency, and recipe execution passes that artifact into the scorer, where it is hashed and audited. ### Pack Content Digest Contract Every claim-facing eval pack summary/datasheet must include `pack_content_sha256`. This is not a cosmetic version string. It is the immutable content identity that lets scoring, audit, and comparison prove that the score still belongs to the same source pack. For recognizer oracle-crop packs, the digest binds each canonical row's sample ID, source image hash, NFC-normalized GT text hash, language, script, domain, GT status, split, and claim eligibility. For page-level OCR packs, the digest binds each canonical page row's sample ID, input image hash, reference artifact hash, reference text hash, reference fields, reference trust metadata, source-document/page/bbox provenance, language, script, domain, GT status, split, and claim eligibility. Claim-safe scoring must fail when the summary digest is missing or stale. Completed score directories must record the recomputed value in `metrics.json` and in the mode-specific run file (`eval-run.json` for recognizer, `page-eval-run.json` for page OCR). Audit must recompute the digest from current source artifacts and reject any mismatch in the run JSON, metrics JSON, or summary/datasheet. Same-task comparison tools must treat the digest as eval-pack identity, so a candidate and baseline cannot be compared as claim-safe if their `pack_content_sha256` values differ. When a test or fixture intentionally edits pack content before scoring, regenerate the summary digest before running the scorer. When content changes after scoring, do not refresh the digest; that is tampering or drift, and audit should fail. For `page_level_ocr` recipes, `eval_pack.eval_manifest` is the canonical page manifest and materialization writes per-page `document.json` files under the root `documents/` directory. External page command templates must write either `{document_json}` directly or write `document.json` below `{output_dir}`. The recipe validator statically audits page manifests for readable source images and hashes, source-document/page/bbox provenance, row claim eligibility, human-review evidence for `human_verified` rows, parseable references, claim-ready reference metadata, and minimum scoring-unit counts overall and per language/script/domain/GT/split slice; any page audit issue or warning blocks claim readiness. For path-complete recognizer packs, the recipe validator also runs `tools/audit_recognizer_eval_pack.py`; any pack-audit issue or warning becomes a claim-readiness blocker, including overall and language/script/domain/GT/split sample-size warnings below the claim minimum. `leakage_sources` must also be parseable and non-empty; page-level leakage manifests must satisfy the page manifest schema, and recognizer label leakage sources must contain valid rows with existing images. A recipe is not claim-ready merely because its label, manifest, summary, script inventory, and image paths exist. For Limbu, keep the pack counts straight. The canonical materialized recognizer pack `outputs/eval-packs/limbu-gorkhapatra-real-v1/` has 1,829 labels and 1,829 manifest rows, not 39. The number 39 belongs to an older operational page run's unresolved review queue and must not be described as the available Limbu eval set. The current Limbu recognizer pack is structurally auditable but not claim-ready: it is machine-converted, has 0 claim-eligible rows, is not declared unseen for selection, and is missing source-document/page/bbox provenance. The separate CDC Maths PDF text-layer pack has 81 silver rendered-line rows, split 69 train / 12 eval; use it for recognizer bootstrap/training experiments, not for claim-safe evaluation. For `page_level_ocr` recipes with `ocr_output_kind=live_detector`, the recipe must also declare `runner.output_audit` bound to `{output_dir}` so page scoring is paired with an operational artifact audit. Limbu page recipes must run `audit-limbu-output` and record strict flags including `--require-capture-prep`, prepared-image quality thresholds (`--min-capture-prepared-width`, `--min-capture-prepared-height`, `--min-capture-prepared-entropy`, `--min-capture-prepared-luminance-stddev`, and `--min-capture-prepared-edge-stddev`), `--require-no-pending-review`, `--require-no-dropped-image-lines`, output quality/volume gates (`--min-line-count`, `--min-average-line-confidence`, and `--min-quality-score`), and the script-specific `--require-script` value for their target, for example `limbu_sirijonga` for Sirijonga-script Limbu or `devanagari_limbu` for Devanagari-written Limbu. This keeps page scoring tied to the operational artifact audit instead of allowing a claim-ready registry entry for an OCR bundle that would fail provenance, capture-quality, output-volume, output-quality, review-resolution, or complete-line-retention gates. Then run the selected recipe: ```bash uv run python tools/run_eval_recipe.py \ --recipes configs/eval-harness-recipes-v1.json \ --recipe-id \ --root . \ --out outputs/evals/ \ --timeout-seconds 3600 \ --min-match-rate 0.99 ``` Add `--require-claim-safe` only when the recipe validator reports no claim blockers and the scorer is expected to emit no warnings. If the recipe uses an external command template, replace the placeholder command in the recipe first; unresolved templates are rejected by default. Recognizer external templates must include `{label_file}`, `{data_dir}`, and `{prediction_file}`; page external templates must include `{input_path}` plus `{document_json}` or `{output_dir}`. Unknown placeholders are validation errors, so command templates must use only the mode-supported placeholders documented by `tools/run_eval_output_command.py`. Recipes that declare `runner.selected_checkpoint_on_test` or non-empty `known_leakage` remain structurally valid but are blocked from claim readiness because the scorer will warn on those same facts at runtime. Runner `checkpoint`, `config`, and `dictionary` values must point to readable local files, not directories, because claim-safe scores record and audit their SHA-256 hashes. Recognizer and page eval-pack summaries must also carry `pack_content_sha256`, the canonical digest over sample ID, input/reference hashes, normalized GT text or structured-reference text, provenance, language, script, domain, GT status, split, and claim eligibility. The scorers recompute this digest from the rows they actually scored; claim-safe runs fail when the summary digest is missing or stale. The recipe runner writes the root-level files a reviewer should open first: `recipe-run.json`, `audit.json`, and `claim-safety.md`. For page-level recipes with `runner.output_audit`, execution runs that audit once per materialized page output directory before scoring and records `output-audit/output-audit-run.json` plus `output-audit/command-template.txt` and Markdown with commands, logs, return codes, and available audit artifact hashes. The runner honors `runner.output_audit.result_filename` when recording the per-page audit JSON; if it is omitted, the default is `limbu-output-audit.json`. The root audit requires each recorded per-page audit JSON to still exist with the recorded SHA-256. It also requires the output-audit command template sidecar when `recipe-run.json` records an output-audit command, rejects a missing or stale sidecar, and replays each per-page output-audit command from that template using the recorded output directory. It also replays the recorded stdout/stderr log SHA-256 values for each per-page audit command. It requires `output-audit/output-audit-run.md` and checks that the Markdown report still reflects the recorded count, per-page output audit rows, log hashes, and recorded audit artifact hashes. The report summarizes what was scored, the GT trust metadata, frozen/unseen declarations, leakage status, micro-CER metrics, output-audit status, warnings, and review artifact locations. The root report is audited against `recipe-run.json`, the score eval-run JSON, and `metrics.json`, with mode-specific expectations for recognizer oracle-crop and page-level OCR score facts, so stale top-level summaries fail audit. The recipe-root audit also checks that `materialize/output-command-run.json` matches the materialization summary embedded in `recipe-run.json`, that `materialize/command-template.txt` and `materialize/output-command-run.md` match the recorded command/output summary, that materialization timestamp, machine platform/ Python metadata, invocation, per-record command, and stdout/stderr log paths and SHA-256 values are present and replay against current log bytes. It also replays materialization record commands from the recorded command template using the recorded recognizer or page materialization fields; if those fields are missing, the root audit fails instead of skipping command replay. For recognizer roots, the root `predictions.txt` path and hash must be the same prediction artifact recorded by the score eval-run JSON. For page roots, each materialization record must carry the source page image path/hash, the recorded hash is rechecked against current bytes, `outputs.documents_dir` must match the score eval-run `documents_dir`, and materialized `document.json` outputs must live under that directory. Recipe-root materialization model/checkpoint/config/dictionary metadata is also checked against the score eval-run metadata. If materialization fails before a score directory exists, keep `materialize/output-command-run-error.json`: it carries the timestamp, machine, invocation, model/checkpoint/config/dictionary metadata, command template, and label or eval-manifest path/hash used for the failed attempt. If the recipe wrapper fails, keep `recipe-run-error.json`: it records the timestamp, machine, command, recipe registry hash, selected recipe validation, root, output directory, exception type, message, and traceback. The underlying recognizer or page score directory also writes its own `claim-safety.md`, so direct scorer runs are reviewable without a recipe wrapper. Audit the finished run directory before reporting the number: ```bash uv run python tools/audit_eval_run.py \ --run-dir outputs/evals/ \ --out outputs/evals//audit.json \ --require-audit-pass ``` The audit recomputes recognizer row edits and aggregate micro-CER from `score/predictions.tsv`; a metric file that is internally ratio-consistent but does not match the emitted GT/prediction rows must fail audit. The run machine metadata must include concrete `platform` and `python` values, and `normalization_profile` must match the scorer-supported `NFC_preserve_whitespace` profile used by the audit recomputation path. The audit also recomputes `normalization_summary` from `predictions.tsv` for recognizer runs and `text-pairs.tsv` for page runs; those fields count raw GT/reference and prediction strings changed by NFC, so normalization-heavy scores remain visible instead of being hidden behind normalized CER. The audit also reconstructs `predictions.tsv` row identity, GT text, prediction text, prediction score, crop hash, language/script/domain/GT/split, and row claim-eligibility fields from the recorded label, prediction, and manifest sources. The same file carries language/script/domain/GT/split, claim-eligibility, and crop image SHA-256 values, so recognizer slice metrics, claim-eligibility provenance, and crop-image identity are recomputed from the emitted row artifact, not trusted as separate aggregates. `worst-lines.csv` is also audited against those canonical prediction rows, including crop hashes, trust metadata, claim eligibility, and source-document/page/bbox provenance, so a failure sample cannot be made more claim-friendly by editing the review artifact after scoring. Recognizer label-line count, scored-line count, matched-line count, unmatched label count, unmatched prediction count, and prediction match rate are recomputed from the recorded source label and prediction files. Recognizer `normalizer_counts` in `metrics.json` and `eval-run.json` are also checked against the recorded summary/datasheet `counts`, so defaulted pack metadata cannot be hidden by editing both score JSON files after scoring. Recognizer and page summary/datasheet files are semantically audited too: if a hash-valid summary declares language, script, domain, GT status, split, eval-pack version, frozen/unseen status, default claim eligibility, or task metadata that contradicts the scored run metadata, the audit fails. Run-level language/script/domain/GT/split lists in `eval-run.json` are also checked against the unique values in `predictions.tsv`. `predictions.tsv` also carries source-document/page/bbox provenance columns, and the audit reconstructs those values from the recorded label/manifest sources. For page runs, the audit also recomputes `line-alignment.csv` row contents from the scored `document.json` and reference JSON, including predicted/reference line IDs, line bboxes, text, IoU, and per-line text CER, then recomputes line detection counts, precision/recall/F1, mean IoU, and aggregate metric means from `line-alignment.csv` and `page-metrics.csv`. The audit also reconstructs `page-metrics.csv` row identity from the eval manifest, resolved `document.json`, embedded `document.source_path`, scorer metadata defaults, and parsed reference fields, so language/script/domain/GT/split, source-document/page/bbox provenance, `claim_eligible`, and `reference_fields` cannot be changed after scoring without failing audit. Run-level language/script/domain/GT/split lists in `page-eval-run.json` must match the unique values in `page-metrics.csv`. `reference-artifacts.tsv` carries the exact reference path, SHA-256, scoreable reference fields, text hash, and reference trust metadata for each scored page, and the audit recomputes it from the eval manifest and reference files. `worst-pages.csv` is audited against `page-metrics.csv` plus `reference-artifacts.tsv`, including input/document hashes, language/script/domain/GT metadata, claim eligibility, source-document/page/bbox provenance, reference hashes, and reference trust fields. `page-metrics.csv` carries raw page text edit/ref counts and exact text-match flags, `text-pairs.tsv` carries the exact prediction/reference page text used for those counts, and the audit recomputes the text-pair artifact from scored `document.json` text plus reference text before rolling it into page text micro-CER, text exact-accuracy totals, and per-language/script/domain/GT/split page slices. It also records source page image and `document.json` SHA-256 values, which the audit rechecks so a page score cannot point at mutated OCR outputs after the fact. Train/dev/eval leakage reports are recomputed from the recorded train manifests and canonical eval rows/manifests, including checked state, row counts, and overlap sets. For page-derived crop training or checkpoint-selection data, include `source_sample_id` on train rows; scorer and audit treat overlap between train `source_sample_id` and eval page `sample_id` as hard leakage even when crop image hashes and line text hashes do not match the full page row directly. Native spot-check calibration summaries embedded in run and metric JSON are checked against the recorded calibration artifact. Metadata contradictions are recomputed from manifest/reference provenance signals and canonical row domain/GT fields. Recognizer provenance coverage and page row metadata coverage counts are recomputed from their source manifests, and page reference coverage is recomputed from `page-metrics.csv`. Page manifest/scored/missing/claim-eligible counts are recomputed from the eval manifest, document outputs, references, and claim-eligibility metadata. Recognizer and page score directories also carry deterministic bootstrap uncertainty summaries in `metrics.json`; the audit recomputes those intervals from `predictions.tsv` or `page-metrics.csv`. Claim-safe runs currently require at least 30 scored rows/pages, so one-page smoke checks and tiny oracle-crop packs must remain non-claim-safe even when their point CER is perfect. Recognizer and page score directories also carry script inventory summaries in `metrics.json` and run JSON; the audit recomputes observed codepoints/graphemes from recognizer GT rows or page reference text and verifies the inventory file hash. A missing inventory file, a stale inventory summary, or missing required codepoints/graphemes remains a claim-safety blocker even when CER is low. They also carry `dictionary_coverage` in `metrics.json`; the audit rehashes the recorded dictionary file, recomputes the non-whitespace GT/reference codepoints, and blocks claim-safe status when the ground truth contains codepoints outside the dictionary vocabulary. `metrics.md` and `claim-safety.md` must surface the same dictionary coverage, normalization-change, and zero-length GT/reference counters, and the audit treats stale Markdown as an output-directory integrity failure. Recognizer metrics record `zero_length_gt_lines`, and page metrics record `zero_length_text_references`; the audit recomputes both from canonical rows and blocks claim-safe status when any scored GT/reference text is empty. The audit also derives claim-safety blockers from durable run facts, so a tampered `claim_safe=true` field without matching warnings still fails if the artifacts show metadata contradictions, a missing manifest or summary/datasheet, an unrun leakage check, leakage overlaps, low recognizer match rate, unsafe GT/domain status, missing provenance, missing uncertainty evidence, missing script inventory evidence, missing required script inventory items, zero-length GT/reference text, too-small claim-facing sample size, recognizer row-level claim ineligibility, or incomplete page coverage. The direct and recipe-root `claim-safety.md` reports are also checked against their JSON run and metric facts, using recognizer or page score fields according to `eval_mode`, including score timestamp, machine, command provenance, hashed checkpoint/config/dictionary artifact identity, normalization-change, dictionary-coverage, script-inventory gaps, metadata contradictions, uncertainty intervals, zero-length GT/reference counters, and per-language/script/domain/GT/split slice metrics, so stale or aggregate-only reviewer-facing summaries fail audit instead of merely having the right section headings. Direct and recipe-root score reports must also list the review artifacts a reviewer needs to open, including `command.txt`, canonical row artifacts, worst-example CSVs, sample-review CSV/HTML files, confusions, and page alignment/reference artifacts. For recipe-root audits, `recipe.json` must still match the selected recipe in the hashed source registry and `recipe-validation.json` must still match `recipe-run.json`; the embedded validation is also recomputed from the current recipe registry and eval-pack audit. Recipe-run timestamp, machine, command, and root output pointers are audited too. Root `audit.json` and the embedded recipe-run audit summary must preserve current score-audit issues, warnings, claim-safe status, and score-dir identity, and the recipe-run score summary must point to and hash the current `score/metrics.json` plus the mode-specific score run file (`score/eval-run.json` for recognizer roots, `score/page-eval-run.json` for page roots). The root train-manifest path/hash list must also match the score eval-run leakage inputs, and the hashed recipe registry must be under the declared recipe root. Do not hand-edit those files after scoring. For page-level Limbu book-candidate recipes that use `tools/score_limbu_book_ocr_candidate.py`, the recipe validator also audits the declared `evidence` chain. It reads the score wrapper, admission report, metrics, and review-coverage report; verifies recorded SHA-256 values; requires champion metrics; requires review-coverage counts for missed and extra lines; and blocks candidates with champion regressions or uncovered review regions. It also parses the recipe `command_template` and requires the scorer command to bind the same `--book-output`, `--book-audit`, `--champion-metrics`, and `--review-coverage-report` paths recorded in the evidence chain. A command string or free-text claim blocker is not enough evidence for current-best registration, and stale evidence cannot be attached to a weaker replay command. The score-level `metrics.md` report is checked against `metrics.json` for core counts, CER values, slices, reference coverage, metric means, and warnings. Recognizer `unmatched.tsv` row identity, label image hashes, prediction scores, and source-level match/count metrics are recomputed from the recorded source label and prediction files, recognizer `confusions.json` is recomputed from canonical predictions, page `confusions.json` is recomputed from canonical `text-pairs.tsv` prediction/reference text, and page `missing.tsv` is recomputed from the eval manifest, document outputs, and reference files. Worst-example CSVs, reviewer CSV order and fields, and reviewer HTML samples are also checked against the canonical scored rows, so stale or hand-edited failure samples do not satisfy reviewability. Completed score artifacts are checked for duplicate scored sample IDs, image paths, document paths, and missing-row keys; duplicate IDs are not only a pre-normalization pack failure. For paper-claim numbers, also add `--require-claim-safe`. ## Recognizer Oracle-Crop Eval Use this manual path for line-crop recognizers only. Do not use it for page-level OCR claims. ### Manual Path 1. Normalize the eval pack into the canonical schema. ```bash uv run python tools/normalize_recognizer_eval_pack.py \ --label-file data//eval.txt \ --data-dir data/ \ --manifest-file data//manifest.jsonl \ --summary-file data//summary.json \ --out outputs/eval-packs/- \ --dataset \ --language \ --script