|
Download ocr-tech/docs/EVAL_HARNESS_RUNBOOK.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 44.4 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/EVAL_HARNESS_RUNBOOK.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/EVAL_HARNESS_RUNBOOK.md
-
curl -L -o EVAL_HARNESS_RUNBOOK.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/EVAL_HARNESS_RUNBOOK.md
44.4 kB
| # Eval Harness Runbook | |
| Updated: 2026-06-20 | |
| This is the operational path for producing a claim-grade OCR eval directory. The | |
| goal is not to get a number quickly; it is to make the number hard to misread. | |
| Each run should leave enough artifacts for a reviewer to answer what was scored, | |
| which model produced it, which ground truth was used, whether the pack was frozen | |
| and unseen for selection, what leakage checks ran, and why the result is or is not | |
| safe for a paper claim. | |
| ## Recipe-First Eval | |
| Use this path when a registered recipe exists for either `recognizer_oracle_crop` | |
| or `page_level_ocr`. Recipes keep validation, leakage checks, output | |
| materialization, scoring, root reports, and audit metadata under one output | |
| directory. | |
| ### Registry Validation | |
| When a recipe exists in `configs/eval-harness-recipes-v1.json`, use it as the | |
| default entry point. This keeps validation, leakage checks, prediction | |
| materialization, and scoring under one output directory. | |
| Before scoring legacy source packs, validate or run the normalization registry: | |
| ```bash | |
| uv run python tools/run_recognizer_pack_normalization.py \ | |
| --config configs/recognizer-eval-pack-normalization-v1.json \ | |
| --root . \ | |
| --all \ | |
| --out outputs/eval-packs/normalization-runs/<run-id> \ | |
| --dry-run \ | |
| --require-valid | |
| ``` | |
| Drop `--dry-run` only when you intentionally want to materialize canonical packs | |
| under `outputs/eval-packs/<pack-version>/`. A materialized pack gets its own | |
| `audit/recognizer-eval-pack-audit.json` and `.md`; read those before wiring the | |
| pack into a claim run. | |
| Then validate intended language/script coverage against the normalization and | |
| eval-run recipe registries: | |
| ```bash | |
| uv run python tools/validate_eval_target_coverage.py \ | |
| --coverage-config configs/eval-target-coverage-v1.json \ | |
| --normalization-config configs/recognizer-eval-pack-normalization-v1.json \ | |
| --eval-recipes configs/eval-harness-recipes-v1.json \ | |
| --root . \ | |
| --out outputs/evals/target-coverage.json \ | |
| --require-valid | |
| ``` | |
| Do not add `--require-coverage-ready` until every target in the selected claim | |
| scope has a normalized eval pack, runnable eval recipe, and no claim blockers. | |
| If a target points at a shared aggregate pack, the validator must show matching | |
| target language/script metadata or an explicit gap; do not use a shared pack | |
| average as evidence for an individual language claim. Coverage target status is | |
| also gated: `eval_status` must be one of the known states, and only | |
| `registered_real_pack` is allowed to become coverage-ready. Use | |
| `partial_real_recognizer_pack`, `shared_devanagari_pack`, or | |
| `missing_real_eval_pack` when a target is deliberately not ready for claims. | |
| For book OCR targets, set `requires_page_eval_recipe` and | |
| `requires_book_candidate_evidence` on the target set or target. That makes coverage | |
| depend on a page-level recipe with book-candidate evidence from the scoring/admission | |
| lane, not just an oracle-crop recognizer score or a loose page replay. The validator | |
| also reports whether the linked book candidate is operationally admitted and | |
| claim-safe, so non-claim-safe Sirijonga and Devanagari book evidence show up as | |
| separate gaps. | |
| ```bash | |
| uv run python tools/validate_eval_harness_recipes.py \ | |
| --recipes configs/eval-harness-recipes-v1.json \ | |
| --root . \ | |
| --require-valid | |
| ``` | |
| Each recipe `eval_pack` must declare `normalization_profile: | |
| NFC_preserve_whitespace`, matching the scorer/auditor recomputation contract, and a | |
| readable, parseable `summary_file` and `script_inventory_file`. The recipe validator | |
| checks the summary JSON object and inventory JSON shape before a recipe is considered | |
| structurally valid, and a summary without dataset/version/eval-pack metadata blocks | |
| claim readiness. Summary/datasheet declarations for language, script, dataset, task, | |
| domain, GT status, split, pack version, frozen/unseen status, and default claim | |
| eligibility must agree with the recipe metadata; contradictions block claim readiness. | |
| Do not paper over a pack that needs another normalization policy; add | |
| scorer support and audit coverage first, then change the recipe. | |
| The script inventory is a JSON object with `required_codepoints` and | |
| `required_graphemes` lists. Codepoints may be written as `U+XXXX` values or literal | |
| single characters; graphemes are literal strings. The scorer records observed coverage | |
| from GT/reference text, and claim-safe runs fail if any required item is missing. | |
| When a runner dictionary is a readable local file, the recipe validator also compares | |
| its non-whitespace codepoints to recognizer GT or page reference text; out-of-vocabulary | |
| GT/reference codepoints block claim readiness before scoring. | |
| If the recipe default or audited row metadata uses `machine_converted_spot_checked` | |
| GT, the recipe must also declare a readable JSON `eval_pack.spotcheck_calibration_file`; | |
| the recipe validator checks its eligibility/recommendation/source-status fields and | |
| language/script/domain consistency, and recipe execution passes that artifact into the | |
| scorer, where it is hashed and audited. | |
| ### Pack Content Digest Contract | |
| Every claim-facing eval pack summary/datasheet must include `pack_content_sha256`. | |
| This is not a cosmetic version string. It is the immutable content identity that lets | |
| scoring, audit, and comparison prove that the score still belongs to the same source | |
| pack. | |
| For recognizer oracle-crop packs, the digest binds each canonical row's sample ID, | |
| source image hash, NFC-normalized GT text hash, language, script, domain, GT status, | |
| split, and claim eligibility. For page-level OCR packs, the digest binds each canonical | |
| page row's sample ID, input image hash, reference artifact hash, reference text hash, | |
| reference fields, reference trust metadata, source-document/page/bbox provenance, | |
| language, script, domain, GT status, split, and claim eligibility. | |
| Claim-safe scoring must fail when the summary digest is missing or stale. Completed | |
| score directories must record the recomputed value in `metrics.json` and in the | |
| mode-specific run file (`eval-run.json` for recognizer, `page-eval-run.json` for page | |
| OCR). Audit must recompute the digest from current source artifacts and reject any | |
| mismatch in the run JSON, metrics JSON, or summary/datasheet. Same-task comparison | |
| tools must treat the digest as eval-pack identity, so a candidate and baseline cannot | |
| be compared as claim-safe if their `pack_content_sha256` values differ. | |
| When a test or fixture intentionally edits pack content before scoring, regenerate the | |
| summary digest before running the scorer. When content changes after scoring, do not | |
| refresh the digest; that is tampering or drift, and audit should fail. | |
| For `page_level_ocr` recipes, `eval_pack.eval_manifest` is the canonical page | |
| manifest and materialization writes per-page `document.json` files under the root | |
| `documents/` directory. External page command templates must write either | |
| `{document_json}` directly or write `document.json` below `{output_dir}`. The recipe | |
| validator statically audits page manifests for readable source images and hashes, | |
| source-document/page/bbox provenance, row claim eligibility, human-review evidence for | |
| `human_verified` rows, parseable references, claim-ready reference metadata, and | |
| minimum scoring-unit counts overall and per language/script/domain/GT/split slice; any | |
| page audit issue or warning blocks claim readiness. For path-complete recognizer packs, | |
| the recipe validator also runs | |
| `tools/audit_recognizer_eval_pack.py`; any pack-audit issue or warning becomes a | |
| claim-readiness blocker, including overall and language/script/domain/GT/split | |
| sample-size warnings below the claim minimum. `leakage_sources` must also be | |
| parseable and non-empty; page-level leakage manifests must satisfy the page manifest | |
| schema, and recognizer label leakage sources must contain valid rows with existing | |
| images. A recipe is not claim-ready merely because its label, manifest, summary, | |
| script inventory, and image paths exist. | |
| For Limbu, keep the pack counts straight. The canonical materialized recognizer pack | |
| `outputs/eval-packs/limbu-gorkhapatra-real-v1/` has 1,829 labels and 1,829 manifest | |
| rows, not 39. The number 39 belongs to an older operational page run's unresolved | |
| review queue and must not be described as the available Limbu eval set. The current | |
| Limbu recognizer pack is structurally auditable but not claim-ready: it is | |
| machine-converted, has 0 claim-eligible rows, is not declared unseen for selection, and | |
| is missing source-document/page/bbox provenance. The separate CDC Maths PDF text-layer | |
| pack has 81 silver rendered-line rows, split 69 train / 12 eval; use it for recognizer | |
| bootstrap/training experiments, not for claim-safe evaluation. | |
| For `page_level_ocr` recipes with `ocr_output_kind=live_detector`, the recipe must | |
| also declare `runner.output_audit` bound to `{output_dir}` so page scoring is paired | |
| with an operational artifact audit. Limbu page recipes must run `audit-limbu-output` | |
| and record strict flags including `--require-capture-prep`, prepared-image quality | |
| thresholds (`--min-capture-prepared-width`, `--min-capture-prepared-height`, | |
| `--min-capture-prepared-entropy`, `--min-capture-prepared-luminance-stddev`, and | |
| `--min-capture-prepared-edge-stddev`), `--require-no-pending-review`, | |
| `--require-no-dropped-image-lines`, output quality/volume gates | |
| (`--min-line-count`, `--min-average-line-confidence`, and `--min-quality-score`), and | |
| the script-specific `--require-script` value for their target, for example | |
| `limbu_sirijonga` for Sirijonga-script Limbu or `devanagari_limbu` for | |
| Devanagari-written Limbu. This keeps page scoring tied to the operational artifact | |
| audit instead of allowing a claim-ready registry entry for an OCR bundle that would | |
| fail provenance, capture-quality, output-volume, output-quality, review-resolution, or | |
| complete-line-retention gates. | |
| Then run the selected recipe: | |
| ```bash | |
| uv run python tools/run_eval_recipe.py \ | |
| --recipes configs/eval-harness-recipes-v1.json \ | |
| --recipe-id <recipe-id> \ | |
| --root . \ | |
| --out outputs/evals/<run-id> \ | |
| --timeout-seconds 3600 \ | |
| --min-match-rate 0.99 | |
| ``` | |
| Add `--require-claim-safe` only when the recipe validator reports no claim | |
| blockers and the scorer is expected to emit no warnings. If the recipe uses an | |
| external command template, replace the placeholder command in the recipe first; | |
| unresolved templates are rejected by default. Recognizer external templates must include | |
| `{label_file}`, `{data_dir}`, and `{prediction_file}`; page external templates must | |
| include `{input_path}` plus `{document_json}` or `{output_dir}`. Unknown placeholders | |
| are validation errors, so command templates must use only the mode-supported | |
| placeholders documented by `tools/run_eval_output_command.py`. Recipes that declare | |
| `runner.selected_checkpoint_on_test` or non-empty `known_leakage` remain structurally | |
| valid but are blocked from claim readiness because the scorer will warn on those same | |
| facts at runtime. Runner `checkpoint`, `config`, and `dictionary` values must point to | |
| readable local files, not directories, because claim-safe scores record and audit their | |
| SHA-256 hashes. | |
| Recognizer and page eval-pack summaries must also carry `pack_content_sha256`, the | |
| canonical digest over sample ID, input/reference hashes, normalized GT text or | |
| structured-reference text, provenance, language, script, domain, GT status, split, and | |
| claim eligibility. The scorers recompute this digest from the rows they actually | |
| scored; claim-safe runs fail when the summary digest is missing or stale. | |
| The recipe runner writes the root-level files a reviewer should open first: | |
| `recipe-run.json`, `audit.json`, and `claim-safety.md`. For page-level recipes with | |
| `runner.output_audit`, execution runs that audit once per materialized page output | |
| directory before scoring and records `output-audit/output-audit-run.json` plus | |
| `output-audit/command-template.txt` and Markdown with commands, logs, return codes, | |
| and available audit artifact hashes. The runner honors | |
| `runner.output_audit.result_filename` when recording the per-page audit JSON; if it is | |
| omitted, the default is `limbu-output-audit.json`. The root audit requires each | |
| recorded per-page audit JSON to still exist with the recorded SHA-256. It also | |
| requires the output-audit command template sidecar when `recipe-run.json` records an | |
| output-audit command, rejects a missing or stale sidecar, and replays each per-page | |
| output-audit command from that template using the recorded output directory. It | |
| also replays the recorded stdout/stderr log SHA-256 values for each per-page audit | |
| command. It requires `output-audit/output-audit-run.md` and checks that the Markdown | |
| report still reflects the recorded count, per-page output audit rows, log hashes, and | |
| recorded audit artifact hashes. | |
| The report summarizes what was scored, the GT trust | |
| metadata, frozen/unseen declarations, leakage status, micro-CER metrics, output-audit | |
| status, warnings, and review artifact locations. The root report is audited against | |
| `recipe-run.json`, the score eval-run JSON, and `metrics.json`, with mode-specific | |
| expectations for recognizer oracle-crop and page-level OCR score facts, so stale | |
| top-level summaries fail audit. | |
| The recipe-root audit also checks that `materialize/output-command-run.json` matches | |
| the materialization summary embedded in `recipe-run.json`, that | |
| `materialize/command-template.txt` and `materialize/output-command-run.md` match the | |
| recorded command/output summary, that materialization timestamp, machine platform/ | |
| Python metadata, invocation, per-record command, and stdout/stderr log paths and | |
| SHA-256 values are present and replay against current log bytes. It also replays | |
| materialization record commands from the recorded command template using the recorded | |
| recognizer or page materialization fields; if those fields are missing, the root audit | |
| fails instead of skipping command replay. For recognizer roots, the root `predictions.txt` path and hash must be the | |
| same prediction artifact recorded by the score eval-run JSON. For page roots, | |
| each materialization record must carry the source page image path/hash, the recorded | |
| hash is rechecked against current bytes, `outputs.documents_dir` must match the score | |
| eval-run `documents_dir`, and materialized `document.json` outputs must live under that directory. Recipe-root materialization | |
| model/checkpoint/config/dictionary metadata is also checked against the score eval-run | |
| metadata. | |
| If materialization fails before a score directory exists, keep | |
| `materialize/output-command-run-error.json`: it carries the timestamp, machine, | |
| invocation, model/checkpoint/config/dictionary metadata, command template, and label | |
| or eval-manifest path/hash used for the failed attempt. If the recipe wrapper fails, | |
| keep `recipe-run-error.json`: it records the timestamp, machine, command, recipe | |
| registry hash, selected recipe validation, root, output directory, exception type, | |
| message, and traceback. | |
| The underlying recognizer or page score directory also writes its own | |
| `claim-safety.md`, so direct scorer runs are reviewable without a recipe wrapper. | |
| Audit the finished run directory before reporting the number: | |
| ```bash | |
| uv run python tools/audit_eval_run.py \ | |
| --run-dir outputs/evals/<run-id> \ | |
| --out outputs/evals/<run-id>/audit.json \ | |
| --require-audit-pass | |
| ``` | |
| The audit recomputes recognizer row edits and aggregate micro-CER from | |
| `score/predictions.tsv`; a metric file that is internally ratio-consistent but does | |
| not match the emitted GT/prediction rows must fail audit. The run machine metadata | |
| must include concrete `platform` and `python` values, and `normalization_profile` | |
| must match the scorer-supported `NFC_preserve_whitespace` profile used by the audit | |
| recomputation path. The audit also recomputes `normalization_summary` from | |
| `predictions.tsv` for recognizer runs and `text-pairs.tsv` for page runs; those | |
| fields count raw GT/reference and prediction strings changed by NFC, so | |
| normalization-heavy scores remain visible instead of being hidden behind normalized | |
| CER. The audit | |
| also reconstructs `predictions.tsv` row identity, GT text, prediction text, | |
| prediction score, crop hash, language/script/domain/GT/split, and row | |
| claim-eligibility fields from the recorded label, prediction, and manifest sources. | |
| The same file carries | |
| language/script/domain/GT/split, claim-eligibility, and crop image SHA-256 values, so | |
| recognizer slice metrics, claim-eligibility provenance, and crop-image identity are | |
| recomputed from the emitted row artifact, not trusted as separate aggregates. | |
| `worst-lines.csv` is also audited against those canonical prediction rows, including | |
| crop hashes, trust metadata, claim eligibility, and source-document/page/bbox | |
| provenance, so a failure sample cannot be made more claim-friendly by editing the | |
| review artifact after scoring. | |
| Recognizer label-line count, scored-line count, matched-line count, unmatched label | |
| count, unmatched prediction count, and prediction match rate are recomputed from the | |
| recorded source label and prediction files. | |
| Recognizer `normalizer_counts` in `metrics.json` and `eval-run.json` are also checked | |
| against the recorded summary/datasheet `counts`, so defaulted pack metadata cannot be | |
| hidden by editing both score JSON files after scoring. | |
| Recognizer and page summary/datasheet files are semantically audited too: if a | |
| hash-valid summary declares language, script, domain, GT status, split, eval-pack | |
| version, frozen/unseen status, default claim eligibility, or task metadata that | |
| contradicts the scored run metadata, the audit fails. | |
| Run-level language/script/domain/GT/split lists in `eval-run.json` are also checked | |
| against the unique values in `predictions.tsv`. | |
| `predictions.tsv` also carries source-document/page/bbox provenance columns, and the | |
| audit reconstructs those values from the recorded label/manifest sources. | |
| For page runs, the audit also recomputes `line-alignment.csv` row contents from the | |
| scored `document.json` and reference JSON, including predicted/reference line IDs, | |
| line bboxes, text, IoU, and per-line text CER, then recomputes line detection counts, | |
| precision/recall/F1, mean IoU, and aggregate metric means from `line-alignment.csv` | |
| and `page-metrics.csv`. | |
| The audit also reconstructs `page-metrics.csv` row identity from the eval manifest, | |
| resolved `document.json`, embedded `document.source_path`, scorer metadata defaults, | |
| and parsed reference fields, so | |
| language/script/domain/GT/split, source-document/page/bbox provenance, | |
| `claim_eligible`, and `reference_fields` cannot be changed after scoring without | |
| failing audit. | |
| Run-level language/script/domain/GT/split lists in `page-eval-run.json` must match | |
| the unique values in `page-metrics.csv`. | |
| `reference-artifacts.tsv` carries the exact reference path, SHA-256, scoreable | |
| reference fields, text hash, and reference trust metadata for each scored page, and | |
| the audit recomputes it from the eval manifest and reference files. | |
| `worst-pages.csv` is audited against `page-metrics.csv` plus | |
| `reference-artifacts.tsv`, including input/document hashes, language/script/domain/GT | |
| metadata, claim eligibility, source-document/page/bbox provenance, reference hashes, | |
| and reference trust fields. | |
| `page-metrics.csv` carries raw page text edit/ref counts and exact text-match flags, | |
| `text-pairs.tsv` carries the exact prediction/reference page text used for those | |
| counts, and the audit recomputes the text-pair artifact from scored `document.json` | |
| text plus reference text before rolling it into page text micro-CER, | |
| text exact-accuracy totals, and per-language/script/domain/GT/split page slices. | |
| It also records source page image and `document.json` SHA-256 values, which the audit | |
| rechecks so a page score cannot point at mutated OCR outputs after the fact. | |
| Train/dev/eval leakage reports are recomputed from the recorded train manifests and | |
| canonical eval rows/manifests, including checked state, row counts, and overlap sets. | |
| For page-derived crop training or checkpoint-selection data, include `source_sample_id` | |
| on train rows; scorer and audit treat overlap between train `source_sample_id` and eval | |
| page `sample_id` as hard leakage even when crop image hashes and line text hashes do not | |
| match the full page row directly. | |
| Native spot-check calibration summaries embedded in run and metric JSON are checked | |
| against the recorded calibration artifact. | |
| Metadata contradictions are recomputed from manifest/reference provenance signals and | |
| canonical row domain/GT fields. | |
| Recognizer provenance coverage and page row metadata coverage counts are recomputed | |
| from their source manifests, and page reference coverage is recomputed from | |
| `page-metrics.csv`. Page manifest/scored/missing/claim-eligible counts are recomputed | |
| from the eval manifest, document outputs, references, and claim-eligibility metadata. | |
| Recognizer and page score directories also carry deterministic bootstrap uncertainty | |
| summaries in `metrics.json`; the audit recomputes those intervals from | |
| `predictions.tsv` or `page-metrics.csv`. Claim-safe runs currently require at least | |
| 30 scored rows/pages, so one-page smoke checks and tiny oracle-crop packs must remain | |
| non-claim-safe even when their point CER is perfect. | |
| Recognizer and page score directories also carry script inventory summaries in | |
| `metrics.json` and run JSON; the audit recomputes observed codepoints/graphemes from | |
| recognizer GT rows or page reference text and verifies the inventory file hash. A | |
| missing inventory file, a stale inventory summary, or missing required | |
| codepoints/graphemes remains a claim-safety blocker even when CER is low. | |
| They also carry `dictionary_coverage` in `metrics.json`; the audit rehashes the | |
| recorded dictionary file, recomputes the non-whitespace GT/reference codepoints, and | |
| blocks claim-safe status when the ground truth contains codepoints outside the | |
| dictionary vocabulary. `metrics.md` and `claim-safety.md` must surface the same | |
| dictionary coverage, normalization-change, and zero-length GT/reference counters, | |
| and the audit treats stale Markdown as an output-directory integrity failure. | |
| Recognizer metrics record `zero_length_gt_lines`, and page metrics record | |
| `zero_length_text_references`; the audit recomputes both from canonical rows and | |
| blocks claim-safe status when any scored GT/reference text is empty. | |
| The audit also derives claim-safety blockers from durable run facts, so a tampered | |
| `claim_safe=true` field without matching warnings still fails if the artifacts show | |
| metadata contradictions, a missing manifest or summary/datasheet, an unrun leakage | |
| check, leakage overlaps, low recognizer match rate, unsafe GT/domain status, missing | |
| provenance, missing uncertainty evidence, missing script inventory evidence, missing | |
| required script inventory items, zero-length GT/reference text, too-small claim-facing | |
| sample size, recognizer row-level claim ineligibility, or incomplete page coverage. | |
| The direct and recipe-root `claim-safety.md` reports are also checked against their | |
| JSON run and metric facts, using recognizer or page score fields according to | |
| `eval_mode`, including score timestamp, machine, command provenance, hashed | |
| checkpoint/config/dictionary artifact identity, normalization-change, | |
| dictionary-coverage, script-inventory gaps, metadata contradictions, uncertainty | |
| intervals, zero-length GT/reference counters, and per-language/script/domain/GT/split | |
| slice metrics, so stale or aggregate-only reviewer-facing summaries fail audit instead | |
| of merely having the right section headings. Direct and recipe-root score reports must | |
| also list the review artifacts a reviewer needs to open, including `command.txt`, | |
| canonical row artifacts, worst-example CSVs, sample-review CSV/HTML files, confusions, | |
| and page alignment/reference artifacts. | |
| For recipe-root audits, `recipe.json` must still match the selected recipe in the | |
| hashed source registry and `recipe-validation.json` must still match | |
| `recipe-run.json`; the embedded validation is also recomputed from the current recipe | |
| registry and eval-pack audit. Recipe-run timestamp, machine, command, and root output | |
| pointers are audited too. Root `audit.json` and the embedded recipe-run audit summary must | |
| preserve current score-audit issues, warnings, claim-safe status, and score-dir | |
| identity, and the recipe-run score summary must point to and hash the current | |
| `score/metrics.json` plus the mode-specific score run file (`score/eval-run.json` | |
| for recognizer roots, `score/page-eval-run.json` for page roots). The root train-manifest | |
| path/hash list must also match the score eval-run leakage inputs, and the hashed | |
| recipe registry must be under the declared recipe root. Do not hand-edit those files | |
| after scoring. | |
| For page-level Limbu book-candidate recipes that use | |
| `tools/score_limbu_book_ocr_candidate.py`, the recipe validator also audits the | |
| declared `evidence` chain. It reads the score wrapper, admission report, metrics, and | |
| review-coverage report; verifies recorded SHA-256 values; requires champion metrics; | |
| requires review-coverage counts for missed and extra lines; and blocks candidates with | |
| champion regressions or uncovered review regions. It also parses the recipe | |
| `command_template` and requires the scorer command to bind the same `--book-output`, | |
| `--book-audit`, `--champion-metrics`, and `--review-coverage-report` paths recorded in | |
| the evidence chain. A command string or free-text claim blocker is not enough evidence | |
| for current-best registration, and stale evidence cannot be attached to a weaker replay | |
| command. | |
| The score-level `metrics.md` report is checked against `metrics.json` for core | |
| counts, CER values, slices, reference coverage, metric means, and warnings. | |
| Recognizer `unmatched.tsv` row identity, label image hashes, prediction scores, and | |
| source-level match/count metrics are recomputed from | |
| the recorded source label and prediction files, recognizer `confusions.json` is | |
| recomputed from canonical predictions, page `confusions.json` is recomputed from | |
| canonical `text-pairs.tsv` prediction/reference text, and page `missing.tsv` is | |
| recomputed from the eval manifest, document outputs, and reference files. | |
| Worst-example CSVs, reviewer CSV order and fields, and reviewer HTML samples are also | |
| checked against the canonical scored rows, so stale or hand-edited failure samples do | |
| not satisfy reviewability. | |
| Completed score artifacts are checked for duplicate scored sample IDs, image paths, | |
| document paths, and missing-row keys; duplicate IDs are not only a pre-normalization | |
| pack failure. | |
| For paper-claim numbers, also add `--require-claim-safe`. | |
| ## Recognizer Oracle-Crop Eval | |
| Use this manual path for line-crop recognizers only. Do not use it for page-level | |
| OCR claims. | |
| ### Manual Path | |
| 1. Normalize the eval pack into the canonical schema. | |
| ```bash | |
| uv run python tools/normalize_recognizer_eval_pack.py \ | |
| --label-file data/<pack>/eval.txt \ | |
| --data-dir data/<pack> \ | |
| --manifest-file data/<pack>/manifest.jsonl \ | |
| --summary-file data/<pack>/summary.json \ | |
| --out outputs/eval-packs/<language>-<pack-version> \ | |
| --dataset <dataset-id> \ | |
| --language <language> \ | |
| --script <script> \ | |
| --domain real_print \ | |
| --gt-status machine_converted \ | |
| --split eval \ | |
| --eval-pack-version <pack-version> \ | |
| --frozen-eval-pack \ | |
| --unseen-for-selection \ | |
| --no-claim-eligible | |
| ``` | |
| If the normalizer reports missing source document, page, bbox, GT status, or claim | |
| eligibility, do not hide it with defaults. Fix the pack or keep the result | |
| non-claim-safe. | |
| If a recognizer manifest row declares an image `sha256`, the scorer verifies that it | |
| matches the current crop bytes. A stale declared hash fails scoring and fails later | |
| audit if an older score directory points at a mutated manifest. | |
| 2. Build train/dev leakage manifests for every model family used to select or | |
| train the checkpoint. | |
| ```bash | |
| uv run python tools/build_recognizer_leakage_manifest.py \ | |
| --label data/<train-pack>/train.txt::data/<train-pack>::train \ | |
| --label data/<train-pack>/eval.txt::data/<train-pack>::dev \ | |
| --out outputs/leakage/<model-family>-train-dev.jsonl \ | |
| --summary outputs/leakage/<model-family>-train-dev.summary.json \ | |
| --source <model-family> \ | |
| --language <language> \ | |
| --script <script> \ | |
| --domain real_print \ | |
| --gt-status machine_converted | |
| ``` | |
| Use `--text-only` only when the corpus intentionally has no images. A text-only | |
| manifest can catch text leakage, but it cannot catch duplicate crop images. | |
| If crops were cut from a page eval candidate, the leakage manifest must preserve the | |
| origin page as `source_sample_id`; otherwise a line-crop pack can look clean under exact | |
| image/text hash checks while still being selected from the eval page. | |
| 3. Materialize predictions through a recorded command. | |
| For repo candidate model cards, omit `--command` and provide `--model-config`: | |
| ```bash | |
| uv run python tools/run_eval_output_command.py \ | |
| --mode recognizer \ | |
| --model-config outputs/<model-package>/model-card.json \ | |
| --out outputs/evals/<run-id>/materialize \ | |
| --system-name <system-name> \ | |
| --model-name <model-name> \ | |
| --checkpoint <local-checkpoint-file> \ | |
| --config <local-training-or-inference-config-file> \ | |
| --dictionary <local-dictionary-file> \ | |
| --label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \ | |
| --data-dir outputs/eval-packs/<language>-<pack-version> \ | |
| --prediction-file outputs/evals/<run-id>/predictions.txt \ | |
| --timeout-seconds 3600 | |
| ``` | |
| For external PaddleOCR/HF runners, pass `--command` explicitly. The command must | |
| write `{prediction_file}` in `image<TAB>prediction<TAB>confidence` format and may | |
| use only `{label_file}`, `{data_dir}`, `{prediction_file}`, and `{out_dir}`. | |
| ```bash | |
| uv run python tools/run_eval_output_command.py \ | |
| --mode recognizer \ | |
| --command 'bash scripts/predict_external_recognizer.sh --labels {label_file} --data {data_dir} --out {prediction_file}' \ | |
| --out outputs/evals/<run-id>/materialize \ | |
| --system-name <system-name> \ | |
| --model-name <model-name> \ | |
| --checkpoint <local-checkpoint-file> \ | |
| --config <local-training-or-inference-config-file> \ | |
| --dictionary <local-dictionary-file> \ | |
| --label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \ | |
| --data-dir outputs/eval-packs/<language>-<pack-version> \ | |
| --prediction-file outputs/evals/<run-id>/predictions.txt \ | |
| --timeout-seconds 3600 | |
| ``` | |
| 4. Score the predictions. | |
| ```bash | |
| uv run python tools/run_recognizer_eval.py \ | |
| --label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \ | |
| --data-dir outputs/eval-packs/<language>-<pack-version> \ | |
| --prediction-file outputs/evals/<run-id>/predictions.txt \ | |
| --out outputs/evals/<run-id>/score \ | |
| --manifest-file outputs/eval-packs/<language>-<pack-version>/manifests/recognizer-eval.jsonl \ | |
| --summary-file outputs/eval-packs/<language>-<pack-version>/summary.json \ | |
| --script-inventory-file outputs/eval-packs/<language>-<pack-version>/script-inventory.json \ | |
| --train-manifest outputs/leakage/<model-family>-train-dev.jsonl \ | |
| --model-name <model-name> \ | |
| --checkpoint <local-checkpoint-file> \ | |
| --config <local-training-or-inference-config-file> \ | |
| --dictionary <local-dictionary-file> \ | |
| --language <language> \ | |
| --script <script> \ | |
| --domain real_print \ | |
| --gt-status machine_converted \ | |
| --split eval \ | |
| --eval-pack-version <pack-version> \ | |
| --frozen-eval-pack \ | |
| --unseen-for-selection \ | |
| --no-selected-checkpoint-on-test \ | |
| --no-claim-eligible \ | |
| --min-match-rate 0.99 | |
| ``` | |
| Only add `--require-claim-safe` when every warning is expected to be absent. If | |
| the GT status is `machine_converted_spot_checked`, also pass | |
| `--spotcheck-calibration-file`. Recipe runs can declare the same artifact under | |
| `eval_pack.spotcheck_calibration_file` so the calibration evidence is part of the | |
| versioned recipe instead of an invisible operator-only flag. | |
| For a claim-safe recognizer run, `--checkpoint`, `--config`, and `--dictionary` | |
| must point to readable local files. The scorer records their SHA-256 hashes in | |
| `eval-run.json`, and `tools/audit_eval_run.py` re-verifies those hashes and checks | |
| that the visible run fields still match the hashed artifact identities. | |
| The scorer also records whether all non-whitespace GT codepoints are present in the | |
| dictionary; missing GT codepoints are a claim-safety warning and are independently | |
| recomputed during audit. | |
| The same audit checks hash-valid summary/datasheet metadata against the direct run's | |
| recorded language/script/domain/GT/split, version, frozen/unseen, default | |
| claim-eligibility, and task metadata. | |
| `--script-inventory-file` must also point to a readable local JSON file; it is hashed in | |
| `eval-run.json`, summarized in `metrics.json`, and audited against the emitted GT rows. | |
| After scoring, audit the score directory: | |
| ```bash | |
| uv run python tools/audit_eval_run.py \ | |
| --run-dir outputs/evals/<run-id>/score \ | |
| --out outputs/evals/<run-id>/score/audit.json \ | |
| --require-audit-pass | |
| ``` | |
| ### Same-Pack Recognizer Comparisons | |
| When a paper claim compares two recognizers, do not compare separate aggregate numbers | |
| by eye. Score both systems on the same eval pack, audit both score directories, then | |
| run the paired comparison tool: | |
| ```bash | |
| uv run python tools/compare_recognizer_eval_runs.py \ | |
| --candidate-score-dir outputs/evals/<candidate-run>/score \ | |
| --baseline-score-dir outputs/evals/<baseline-run>/score \ | |
| --candidate-name <candidate-name> \ | |
| --baseline-name <baseline-name> \ | |
| --out outputs/evals/<comparison-run> \ | |
| --min-pairs 30 \ | |
| --require-claim-safe | |
| ``` | |
| The tool pairs rows by `sample_id`, checks that eval-pack identity including | |
| `pack_content_sha256` and row GT identity match, and emits `comparison.json`, | |
| `comparison.md`, `paired-rows.tsv`, and | |
| `command.txt`. It reports codepoint and grapheme micro-CER deltas, paired | |
| wins/losses/ties, deterministic paired-row bootstrap confidence intervals for the | |
| improvement, and a two-sided exact sign-test p-value. A comparison is claim-safe only | |
| when both input score directories are claim-safe, the rows are fully paired on the same | |
| eval pack, the bootstrap lower bound is above zero, the sign test passes the configured | |
| alpha, the candidate actually improves the baseline, and comparison timestamp/machine/ | |
| command provenance is present and internally consistent. | |
| Audit the comparison directory before using it in a claim: | |
| ```bash | |
| uv run python tools/compare_recognizer_eval_runs.py \ | |
| --audit-only \ | |
| --comparison-dir outputs/evals/<comparison-run> \ | |
| --require-claim-safe | |
| ``` | |
| For page-level OCR system comparisons, use the page comparison tool instead of the | |
| recognizer comparison tool: | |
| ```bash | |
| uv run python tools/compare_page_ocr_eval_runs.py \ | |
| --candidate-score-dir outputs/evals/<candidate-page-run> \ | |
| --baseline-score-dir outputs/evals/<baseline-page-run> \ | |
| --candidate-name <candidate-name> \ | |
| --baseline-name <baseline-name> \ | |
| --out outputs/evals/<page-comparison-run> \ | |
| --min-pairs 30 \ | |
| --require-claim-safe | |
| ``` | |
| It pairs pages by `sample_id`, checks page eval-pack identity including | |
| `pack_content_sha256` plus page/reference text identity, and emits `comparison.json`, | |
| `comparison.md`, `paired-pages.tsv`, and `command.txt` for page text | |
| codepoint/grapheme CER deltas, paired wins/losses/ties, paired-page bootstrap | |
| intervals, and a two-sided exact sign-test p-value. Audit it the same way: | |
| ```bash | |
| uv run python tools/compare_page_ocr_eval_runs.py \ | |
| --audit-only \ | |
| --comparison-dir outputs/evals/<page-comparison-run> \ | |
| --require-claim-safe | |
| ``` | |
| ## Native-Reader Spot Check | |
| Machine-converted labels may be scored, but they are not human-verified evidence. | |
| Use this path to create auditable calibration evidence. | |
| ```bash | |
| uv run python tools/prepare_native_spotcheck_bundle.py \ | |
| --label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \ | |
| --data-dir outputs/eval-packs/<language>-<pack-version> \ | |
| --manifest-file outputs/eval-packs/<language>-<pack-version>/manifests/recognizer-eval.jsonl \ | |
| --prediction-file outputs/evals/<run-id>/predictions.txt \ | |
| --out outputs/spotchecks/<language>-<pack-version> \ | |
| --sample-size 50 \ | |
| --seed 20260619 \ | |
| --title '<language> native-reader spot check' | |
| ``` | |
| After a reviewer exports the completed CSV: | |
| ```bash | |
| uv run python tools/apply_native_spotcheck_reviews.py \ | |
| --review-file outputs/spotchecks/<language>-<pack-version>/reviewed-spotcheck.csv \ | |
| --out outputs/spotchecks/<language>-<pack-version>/calibration \ | |
| --language <language> \ | |
| --script <script> \ | |
| --domain real_print \ | |
| --source-gt-status machine_converted \ | |
| --min-usable-rows 30 \ | |
| --max-machine-grapheme-cer 0.02 \ | |
| --max-unusable-rate 0.05 | |
| ``` | |
| If the calibration output says `claim_evidence_eligible=false`, do not mark the | |
| pack `machine_converted_spot_checked`. Completed rows without reviewer identity | |
| are not eligible calibration evidence; fix the review export rather than | |
| overriding the gate. The scorer also requires the calibration artifact's language, | |
| script, domain, `source_gt_status=machine_converted`, and | |
| `recommended_pack_gt_status=machine_converted_spot_checked` to match the scored run. | |
| ## Page-Level OCR Eval | |
| Use this path for end-to-end page claims. Recognizer-only scores cannot support a | |
| page OCR claim because detection, crop lineage, and reading order are absent. | |
| 1. Materialize page outputs through a recorded command. | |
| ```bash | |
| uv run python tools/run_eval_output_command.py \ | |
| --mode page \ | |
| --model-config outputs/<page-model-package>/model-card.json \ | |
| --out outputs/page-evals/<run-id>/materialize \ | |
| --system-name <system-name> \ | |
| --model-name <model-name> \ | |
| --checkpoint <local-checkpoint-file> \ | |
| --config <local-training-or-inference-config-file> \ | |
| --dictionary <local-dictionary-file> \ | |
| --eval-manifest data/<page-eval-pack>/manifests/eval.jsonl \ | |
| --documents-dir outputs/page-evals/<run-id>/documents \ | |
| --timeout-seconds 3600 | |
| ``` | |
| For an external runner, pass `--command` and write `{document_json}` for each | |
| input page. Page commands may use only `{input_path}`, `{sample_id}`, `{output_dir}`, | |
| `{document_json}`, `{documents_dir}`, and `{out_dir}`. | |
| 2. Score saved `document.json` outputs. | |
| ```bash | |
| uv run python tools/run_page_ocr_eval.py \ | |
| --eval-manifest data/<page-eval-pack>/manifests/eval.jsonl \ | |
| --documents-dir outputs/page-evals/<run-id>/documents \ | |
| --ocr-output-kind live_detector \ | |
| --out outputs/page-evals/<run-id>/score \ | |
| --summary-file data/<page-eval-pack>/summary.json \ | |
| --script-inventory-file data/<page-eval-pack>/script-inventory.json \ | |
| --model-name <model-name> \ | |
| --checkpoint <local-checkpoint-file> \ | |
| --config <local-training-or-inference-config-file> \ | |
| --dictionary <local-dictionary-file> \ | |
| --eval-pack-version <pack-version> \ | |
| --frozen-eval-pack \ | |
| --unseen-for-selection \ | |
| --no-selected-checkpoint-on-test \ | |
| --train-manifest outputs/leakage/<model-family>-train-dev.jsonl | |
| ``` | |
| For direct live-detector page scoring, claim-safe mode requires a passing output audit | |
| for every scored page output directory. By default the scorer looks for | |
| `limbu-output-audit.json` with `passed=true` beside each `document.json`; use | |
| `--require-output-audit`, `--output-audit-filename`, and | |
| `--output-audit-passed-field` when scoring a different page-output audit format. | |
| The page auditor replays those per-page audit artifacts from `page-metrics.csv` and | |
| fails if a recorded `page_output_audit` summary no longer matches the current files, | |
| so deleting or editing an output audit after scoring invalidates the score directory. | |
| A paper-safe page claim requires reference text plus page/line layout references | |
| that let the scorer report detection/alignment/reading-order behavior. If line | |
| bboxes are absent, the result is not a full page-level OCR claim. Each page | |
| manifest row must also carry explicit `language`, `script`, `domain`, and | |
| `gt_status` metadata plus source-document, page, bbox, and claim-eligibility | |
| provenance; CLI defaults are recorded as fallbacks, not as row-level claim evidence. | |
| The page summary must carry a `pack_content_sha256` over the canonical page row, | |
| input-image, reference-artifact, reference-text, and provenance identity; claim-safe | |
| scoring rejects summaries whose digest is missing or stale. | |
| Page manifest `sample_id` and `image_path` values must be unique, | |
| and any declared row `sha256` must match the actual input image bytes; duplicate IDs, | |
| duplicate input images, or stale image hashes fail before scoring. If page GT status | |
| is `machine_converted_spot_checked`, pass | |
| `--spotcheck-calibration-file`; without an eligible calibration artifact the | |
| page result is not claim-safe. | |
| For a claim-safe page run, `--checkpoint`, `--config`, and `--dictionary` must point | |
| to readable local files whose SHA-256 hashes can be recorded and audited. | |
| `--script-inventory-file` must point to a readable local JSON file; the scorer records | |
| which required codepoints/graphemes were covered by reference text and the auditor | |
| recomputes that summary from the eval manifest and references. | |
| `--ocr-output-kind` must identify what kind of document outputs were scored. Use | |
| `live_detector` for normal end-to-end page OCR outputs, `oracle_line_boxes` for | |
| diagnostic runs where reference/PDF line boxes are fed to recognizers, and | |
| `sidecar_fixture` only for tests or synthetic fixtures. For Limbu, the detector | |
| diagnostic requires scoring the same frozen page pack once as `live_detector` and once | |
| as `oracle_line_boxes`, then comparing the two score directories with | |
| `tools/compare_page_ocr_eval_runs.py`. | |
| Audit the page score directory: | |
| ```bash | |
| uv run python tools/audit_eval_run.py \ | |
| --run-dir outputs/page-evals/<run-id>/score \ | |
| --out outputs/page-evals/<run-id>/score/audit.json \ | |
| --require-audit-pass | |
| ``` | |
| ## Claim-Safe Gates | |
| A recognizer or page result is not paper-safe unless all of these are true: | |
| - Eval pack is frozen, versioned, and declared unseen for model/checkpoint | |
| selection. | |
| - The score records a manifest and summary/datasheet for the eval pack. | |
| - The score records a script inventory file, its hash, observed codepoint/grapheme | |
| coverage, and no missing required codepoints or graphemes. | |
| - Manifest row IDs and input image paths are unique, recognizer manifest image | |
| paths/basenames cannot collide with basename-based prediction matching, and declared | |
| source-image hashes match the current files; stale recognizer crop hashes, duplicate | |
| page manifest IDs or image paths, stale page manifest image hashes, and duplicate | |
| completed score artifact IDs fail audit. | |
| - Every scored row has source-document, page, bbox, language, script, domain, | |
| split, GT status, and claim eligibility metadata, and every scored row is claim | |
| eligible. | |
| - GT status is `human_verified` or backed by eligible native-reader calibration | |
| as `machine_converted_spot_checked`. | |
| - Rows declared `human_verified` carry human review evidence such as `reviewed_by`, | |
| `review_artifact`, double-review, agreement, or correction-rate metadata; the trust | |
| label alone is not claim evidence. | |
| - The domain is real target material, not synthetic or transliterated evidence. | |
| - Row/reference provenance does not contradict the declared domain or GT status: | |
| synthetic/transliterated provenance cannot be claimed as real material, and | |
| machine-converted provenance cannot be claimed as `human_verified`. | |
| - Train/dev leakage manifests were passed and checked. | |
| - Model artifacts are hash-verifiable: recognizer and page runs require checkpoint, | |
| config, and dictionary files, and | |
| the run fields must match the hashed artifact identity entries. | |
| - Model identity is recorded: recognizer and page runs require a non-empty model name, | |
| and page runs require a non-empty system name. | |
| - No hard leakage exists by sample ID, image hash, image path, or combined | |
| sample hash. | |
| - The prediction match rate is at least the configured threshold. | |
| - No scored recognizer GT row or page text reference is zero-length. | |
| - Headline reporting is per language/script/domain/GT status, not a pooled | |
| average hiding weak languages, and every emitted language/script/domain/GT/split | |
| slice has at least the claim-facing minimum scoring units. | |
| - Same-pack recognizer comparisons use `tools/compare_recognizer_eval_runs.py` and pass | |
| its paired bootstrap/sign-test gates; separate aggregate score directories are not | |
| enough evidence for a comparative claim. | |
| - The score directory contains `claim-safety.md` with `What Was Scored`, `Ground | |
| Truth And Pack`, `Metrics`, and `Warnings` sections. | |