ocr-projects-archive / ocr-tech /docs /EVAL_HARNESS_RUNBOOK.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
44.4 kB
# Eval Harness Runbook
Updated: 2026-06-20
This is the operational path for producing a claim-grade OCR eval directory. The
goal is not to get a number quickly; it is to make the number hard to misread.
Each run should leave enough artifacts for a reviewer to answer what was scored,
which model produced it, which ground truth was used, whether the pack was frozen
and unseen for selection, what leakage checks ran, and why the result is or is not
safe for a paper claim.
## Recipe-First Eval
Use this path when a registered recipe exists for either `recognizer_oracle_crop`
or `page_level_ocr`. Recipes keep validation, leakage checks, output
materialization, scoring, root reports, and audit metadata under one output
directory.
### Registry Validation
When a recipe exists in `configs/eval-harness-recipes-v1.json`, use it as the
default entry point. This keeps validation, leakage checks, prediction
materialization, and scoring under one output directory.
Before scoring legacy source packs, validate or run the normalization registry:
```bash
uv run python tools/run_recognizer_pack_normalization.py \
--config configs/recognizer-eval-pack-normalization-v1.json \
--root . \
--all \
--out outputs/eval-packs/normalization-runs/<run-id> \
--dry-run \
--require-valid
```
Drop `--dry-run` only when you intentionally want to materialize canonical packs
under `outputs/eval-packs/<pack-version>/`. A materialized pack gets its own
`audit/recognizer-eval-pack-audit.json` and `.md`; read those before wiring the
pack into a claim run.
Then validate intended language/script coverage against the normalization and
eval-run recipe registries:
```bash
uv run python tools/validate_eval_target_coverage.py \
--coverage-config configs/eval-target-coverage-v1.json \
--normalization-config configs/recognizer-eval-pack-normalization-v1.json \
--eval-recipes configs/eval-harness-recipes-v1.json \
--root . \
--out outputs/evals/target-coverage.json \
--require-valid
```
Do not add `--require-coverage-ready` until every target in the selected claim
scope has a normalized eval pack, runnable eval recipe, and no claim blockers.
If a target points at a shared aggregate pack, the validator must show matching
target language/script metadata or an explicit gap; do not use a shared pack
average as evidence for an individual language claim. Coverage target status is
also gated: `eval_status` must be one of the known states, and only
`registered_real_pack` is allowed to become coverage-ready. Use
`partial_real_recognizer_pack`, `shared_devanagari_pack`, or
`missing_real_eval_pack` when a target is deliberately not ready for claims.
For book OCR targets, set `requires_page_eval_recipe` and
`requires_book_candidate_evidence` on the target set or target. That makes coverage
depend on a page-level recipe with book-candidate evidence from the scoring/admission
lane, not just an oracle-crop recognizer score or a loose page replay. The validator
also reports whether the linked book candidate is operationally admitted and
claim-safe, so non-claim-safe Sirijonga and Devanagari book evidence show up as
separate gaps.
```bash
uv run python tools/validate_eval_harness_recipes.py \
--recipes configs/eval-harness-recipes-v1.json \
--root . \
--require-valid
```
Each recipe `eval_pack` must declare `normalization_profile:
NFC_preserve_whitespace`, matching the scorer/auditor recomputation contract, and a
readable, parseable `summary_file` and `script_inventory_file`. The recipe validator
checks the summary JSON object and inventory JSON shape before a recipe is considered
structurally valid, and a summary without dataset/version/eval-pack metadata blocks
claim readiness. Summary/datasheet declarations for language, script, dataset, task,
domain, GT status, split, pack version, frozen/unseen status, and default claim
eligibility must agree with the recipe metadata; contradictions block claim readiness.
Do not paper over a pack that needs another normalization policy; add
scorer support and audit coverage first, then change the recipe.
The script inventory is a JSON object with `required_codepoints` and
`required_graphemes` lists. Codepoints may be written as `U+XXXX` values or literal
single characters; graphemes are literal strings. The scorer records observed coverage
from GT/reference text, and claim-safe runs fail if any required item is missing.
When a runner dictionary is a readable local file, the recipe validator also compares
its non-whitespace codepoints to recognizer GT or page reference text; out-of-vocabulary
GT/reference codepoints block claim readiness before scoring.
If the recipe default or audited row metadata uses `machine_converted_spot_checked`
GT, the recipe must also declare a readable JSON `eval_pack.spotcheck_calibration_file`;
the recipe validator checks its eligibility/recommendation/source-status fields and
language/script/domain consistency, and recipe execution passes that artifact into the
scorer, where it is hashed and audited.
### Pack Content Digest Contract
Every claim-facing eval pack summary/datasheet must include `pack_content_sha256`.
This is not a cosmetic version string. It is the immutable content identity that lets
scoring, audit, and comparison prove that the score still belongs to the same source
pack.
For recognizer oracle-crop packs, the digest binds each canonical row's sample ID,
source image hash, NFC-normalized GT text hash, language, script, domain, GT status,
split, and claim eligibility. For page-level OCR packs, the digest binds each canonical
page row's sample ID, input image hash, reference artifact hash, reference text hash,
reference fields, reference trust metadata, source-document/page/bbox provenance,
language, script, domain, GT status, split, and claim eligibility.
Claim-safe scoring must fail when the summary digest is missing or stale. Completed
score directories must record the recomputed value in `metrics.json` and in the
mode-specific run file (`eval-run.json` for recognizer, `page-eval-run.json` for page
OCR). Audit must recompute the digest from current source artifacts and reject any
mismatch in the run JSON, metrics JSON, or summary/datasheet. Same-task comparison
tools must treat the digest as eval-pack identity, so a candidate and baseline cannot
be compared as claim-safe if their `pack_content_sha256` values differ.
When a test or fixture intentionally edits pack content before scoring, regenerate the
summary digest before running the scorer. When content changes after scoring, do not
refresh the digest; that is tampering or drift, and audit should fail.
For `page_level_ocr` recipes, `eval_pack.eval_manifest` is the canonical page
manifest and materialization writes per-page `document.json` files under the root
`documents/` directory. External page command templates must write either
`{document_json}` directly or write `document.json` below `{output_dir}`. The recipe
validator statically audits page manifests for readable source images and hashes,
source-document/page/bbox provenance, row claim eligibility, human-review evidence for
`human_verified` rows, parseable references, claim-ready reference metadata, and
minimum scoring-unit counts overall and per language/script/domain/GT/split slice; any
page audit issue or warning blocks claim readiness. For path-complete recognizer packs,
the recipe validator also runs
`tools/audit_recognizer_eval_pack.py`; any pack-audit issue or warning becomes a
claim-readiness blocker, including overall and language/script/domain/GT/split
sample-size warnings below the claim minimum. `leakage_sources` must also be
parseable and non-empty; page-level leakage manifests must satisfy the page manifest
schema, and recognizer label leakage sources must contain valid rows with existing
images. A recipe is not claim-ready merely because its label, manifest, summary,
script inventory, and image paths exist.
For Limbu, keep the pack counts straight. The canonical materialized recognizer pack
`outputs/eval-packs/limbu-gorkhapatra-real-v1/` has 1,829 labels and 1,829 manifest
rows, not 39. The number 39 belongs to an older operational page run's unresolved
review queue and must not be described as the available Limbu eval set. The current
Limbu recognizer pack is structurally auditable but not claim-ready: it is
machine-converted, has 0 claim-eligible rows, is not declared unseen for selection, and
is missing source-document/page/bbox provenance. The separate CDC Maths PDF text-layer
pack has 81 silver rendered-line rows, split 69 train / 12 eval; use it for recognizer
bootstrap/training experiments, not for claim-safe evaluation.
For `page_level_ocr` recipes with `ocr_output_kind=live_detector`, the recipe must
also declare `runner.output_audit` bound to `{output_dir}` so page scoring is paired
with an operational artifact audit. Limbu page recipes must run `audit-limbu-output`
and record strict flags including `--require-capture-prep`, prepared-image quality
thresholds (`--min-capture-prepared-width`, `--min-capture-prepared-height`,
`--min-capture-prepared-entropy`, `--min-capture-prepared-luminance-stddev`, and
`--min-capture-prepared-edge-stddev`), `--require-no-pending-review`,
`--require-no-dropped-image-lines`, output quality/volume gates
(`--min-line-count`, `--min-average-line-confidence`, and `--min-quality-score`), and
the script-specific `--require-script` value for their target, for example
`limbu_sirijonga` for Sirijonga-script Limbu or `devanagari_limbu` for
Devanagari-written Limbu. This keeps page scoring tied to the operational artifact
audit instead of allowing a claim-ready registry entry for an OCR bundle that would
fail provenance, capture-quality, output-volume, output-quality, review-resolution, or
complete-line-retention gates.
Then run the selected recipe:
```bash
uv run python tools/run_eval_recipe.py \
--recipes configs/eval-harness-recipes-v1.json \
--recipe-id <recipe-id> \
--root . \
--out outputs/evals/<run-id> \
--timeout-seconds 3600 \
--min-match-rate 0.99
```
Add `--require-claim-safe` only when the recipe validator reports no claim
blockers and the scorer is expected to emit no warnings. If the recipe uses an
external command template, replace the placeholder command in the recipe first;
unresolved templates are rejected by default. Recognizer external templates must include
`{label_file}`, `{data_dir}`, and `{prediction_file}`; page external templates must
include `{input_path}` plus `{document_json}` or `{output_dir}`. Unknown placeholders
are validation errors, so command templates must use only the mode-supported
placeholders documented by `tools/run_eval_output_command.py`. Recipes that declare
`runner.selected_checkpoint_on_test` or non-empty `known_leakage` remain structurally
valid but are blocked from claim readiness because the scorer will warn on those same
facts at runtime. Runner `checkpoint`, `config`, and `dictionary` values must point to
readable local files, not directories, because claim-safe scores record and audit their
SHA-256 hashes.
Recognizer and page eval-pack summaries must also carry `pack_content_sha256`, the
canonical digest over sample ID, input/reference hashes, normalized GT text or
structured-reference text, provenance, language, script, domain, GT status, split, and
claim eligibility. The scorers recompute this digest from the rows they actually
scored; claim-safe runs fail when the summary digest is missing or stale.
The recipe runner writes the root-level files a reviewer should open first:
`recipe-run.json`, `audit.json`, and `claim-safety.md`. For page-level recipes with
`runner.output_audit`, execution runs that audit once per materialized page output
directory before scoring and records `output-audit/output-audit-run.json` plus
`output-audit/command-template.txt` and Markdown with commands, logs, return codes,
and available audit artifact hashes. The runner honors
`runner.output_audit.result_filename` when recording the per-page audit JSON; if it is
omitted, the default is `limbu-output-audit.json`. The root audit requires each
recorded per-page audit JSON to still exist with the recorded SHA-256. It also
requires the output-audit command template sidecar when `recipe-run.json` records an
output-audit command, rejects a missing or stale sidecar, and replays each per-page
output-audit command from that template using the recorded output directory. It
also replays the recorded stdout/stderr log SHA-256 values for each per-page audit
command. It requires `output-audit/output-audit-run.md` and checks that the Markdown
report still reflects the recorded count, per-page output audit rows, log hashes, and
recorded audit artifact hashes.
The report summarizes what was scored, the GT trust
metadata, frozen/unseen declarations, leakage status, micro-CER metrics, output-audit
status, warnings, and review artifact locations. The root report is audited against
`recipe-run.json`, the score eval-run JSON, and `metrics.json`, with mode-specific
expectations for recognizer oracle-crop and page-level OCR score facts, so stale
top-level summaries fail audit.
The recipe-root audit also checks that `materialize/output-command-run.json` matches
the materialization summary embedded in `recipe-run.json`, that
`materialize/command-template.txt` and `materialize/output-command-run.md` match the
recorded command/output summary, that materialization timestamp, machine platform/
Python metadata, invocation, per-record command, and stdout/stderr log paths and
SHA-256 values are present and replay against current log bytes. It also replays
materialization record commands from the recorded command template using the recorded
recognizer or page materialization fields; if those fields are missing, the root audit
fails instead of skipping command replay. For recognizer roots, the root `predictions.txt` path and hash must be the
same prediction artifact recorded by the score eval-run JSON. For page roots,
each materialization record must carry the source page image path/hash, the recorded
hash is rechecked against current bytes, `outputs.documents_dir` must match the score
eval-run `documents_dir`, and materialized `document.json` outputs must live under that directory. Recipe-root materialization
model/checkpoint/config/dictionary metadata is also checked against the score eval-run
metadata.
If materialization fails before a score directory exists, keep
`materialize/output-command-run-error.json`: it carries the timestamp, machine,
invocation, model/checkpoint/config/dictionary metadata, command template, and label
or eval-manifest path/hash used for the failed attempt. If the recipe wrapper fails,
keep `recipe-run-error.json`: it records the timestamp, machine, command, recipe
registry hash, selected recipe validation, root, output directory, exception type,
message, and traceback.
The underlying recognizer or page score directory also writes its own
`claim-safety.md`, so direct scorer runs are reviewable without a recipe wrapper.
Audit the finished run directory before reporting the number:
```bash
uv run python tools/audit_eval_run.py \
--run-dir outputs/evals/<run-id> \
--out outputs/evals/<run-id>/audit.json \
--require-audit-pass
```
The audit recomputes recognizer row edits and aggregate micro-CER from
`score/predictions.tsv`; a metric file that is internally ratio-consistent but does
not match the emitted GT/prediction rows must fail audit. The run machine metadata
must include concrete `platform` and `python` values, and `normalization_profile`
must match the scorer-supported `NFC_preserve_whitespace` profile used by the audit
recomputation path. The audit also recomputes `normalization_summary` from
`predictions.tsv` for recognizer runs and `text-pairs.tsv` for page runs; those
fields count raw GT/reference and prediction strings changed by NFC, so
normalization-heavy scores remain visible instead of being hidden behind normalized
CER. The audit
also reconstructs `predictions.tsv` row identity, GT text, prediction text,
prediction score, crop hash, language/script/domain/GT/split, and row
claim-eligibility fields from the recorded label, prediction, and manifest sources.
The same file carries
language/script/domain/GT/split, claim-eligibility, and crop image SHA-256 values, so
recognizer slice metrics, claim-eligibility provenance, and crop-image identity are
recomputed from the emitted row artifact, not trusted as separate aggregates.
`worst-lines.csv` is also audited against those canonical prediction rows, including
crop hashes, trust metadata, claim eligibility, and source-document/page/bbox
provenance, so a failure sample cannot be made more claim-friendly by editing the
review artifact after scoring.
Recognizer label-line count, scored-line count, matched-line count, unmatched label
count, unmatched prediction count, and prediction match rate are recomputed from the
recorded source label and prediction files.
Recognizer `normalizer_counts` in `metrics.json` and `eval-run.json` are also checked
against the recorded summary/datasheet `counts`, so defaulted pack metadata cannot be
hidden by editing both score JSON files after scoring.
Recognizer and page summary/datasheet files are semantically audited too: if a
hash-valid summary declares language, script, domain, GT status, split, eval-pack
version, frozen/unseen status, default claim eligibility, or task metadata that
contradicts the scored run metadata, the audit fails.
Run-level language/script/domain/GT/split lists in `eval-run.json` are also checked
against the unique values in `predictions.tsv`.
`predictions.tsv` also carries source-document/page/bbox provenance columns, and the
audit reconstructs those values from the recorded label/manifest sources.
For page runs, the audit also recomputes `line-alignment.csv` row contents from the
scored `document.json` and reference JSON, including predicted/reference line IDs,
line bboxes, text, IoU, and per-line text CER, then recomputes line detection counts,
precision/recall/F1, mean IoU, and aggregate metric means from `line-alignment.csv`
and `page-metrics.csv`.
The audit also reconstructs `page-metrics.csv` row identity from the eval manifest,
resolved `document.json`, embedded `document.source_path`, scorer metadata defaults,
and parsed reference fields, so
language/script/domain/GT/split, source-document/page/bbox provenance,
`claim_eligible`, and `reference_fields` cannot be changed after scoring without
failing audit.
Run-level language/script/domain/GT/split lists in `page-eval-run.json` must match
the unique values in `page-metrics.csv`.
`reference-artifacts.tsv` carries the exact reference path, SHA-256, scoreable
reference fields, text hash, and reference trust metadata for each scored page, and
the audit recomputes it from the eval manifest and reference files.
`worst-pages.csv` is audited against `page-metrics.csv` plus
`reference-artifacts.tsv`, including input/document hashes, language/script/domain/GT
metadata, claim eligibility, source-document/page/bbox provenance, reference hashes,
and reference trust fields.
`page-metrics.csv` carries raw page text edit/ref counts and exact text-match flags,
`text-pairs.tsv` carries the exact prediction/reference page text used for those
counts, and the audit recomputes the text-pair artifact from scored `document.json`
text plus reference text before rolling it into page text micro-CER,
text exact-accuracy totals, and per-language/script/domain/GT/split page slices.
It also records source page image and `document.json` SHA-256 values, which the audit
rechecks so a page score cannot point at mutated OCR outputs after the fact.
Train/dev/eval leakage reports are recomputed from the recorded train manifests and
canonical eval rows/manifests, including checked state, row counts, and overlap sets.
For page-derived crop training or checkpoint-selection data, include `source_sample_id`
on train rows; scorer and audit treat overlap between train `source_sample_id` and eval
page `sample_id` as hard leakage even when crop image hashes and line text hashes do not
match the full page row directly.
Native spot-check calibration summaries embedded in run and metric JSON are checked
against the recorded calibration artifact.
Metadata contradictions are recomputed from manifest/reference provenance signals and
canonical row domain/GT fields.
Recognizer provenance coverage and page row metadata coverage counts are recomputed
from their source manifests, and page reference coverage is recomputed from
`page-metrics.csv`. Page manifest/scored/missing/claim-eligible counts are recomputed
from the eval manifest, document outputs, references, and claim-eligibility metadata.
Recognizer and page score directories also carry deterministic bootstrap uncertainty
summaries in `metrics.json`; the audit recomputes those intervals from
`predictions.tsv` or `page-metrics.csv`. Claim-safe runs currently require at least
30 scored rows/pages, so one-page smoke checks and tiny oracle-crop packs must remain
non-claim-safe even when their point CER is perfect.
Recognizer and page score directories also carry script inventory summaries in
`metrics.json` and run JSON; the audit recomputes observed codepoints/graphemes from
recognizer GT rows or page reference text and verifies the inventory file hash. A
missing inventory file, a stale inventory summary, or missing required
codepoints/graphemes remains a claim-safety blocker even when CER is low.
They also carry `dictionary_coverage` in `metrics.json`; the audit rehashes the
recorded dictionary file, recomputes the non-whitespace GT/reference codepoints, and
blocks claim-safe status when the ground truth contains codepoints outside the
dictionary vocabulary. `metrics.md` and `claim-safety.md` must surface the same
dictionary coverage, normalization-change, and zero-length GT/reference counters,
and the audit treats stale Markdown as an output-directory integrity failure.
Recognizer metrics record `zero_length_gt_lines`, and page metrics record
`zero_length_text_references`; the audit recomputes both from canonical rows and
blocks claim-safe status when any scored GT/reference text is empty.
The audit also derives claim-safety blockers from durable run facts, so a tampered
`claim_safe=true` field without matching warnings still fails if the artifacts show
metadata contradictions, a missing manifest or summary/datasheet, an unrun leakage
check, leakage overlaps, low recognizer match rate, unsafe GT/domain status, missing
provenance, missing uncertainty evidence, missing script inventory evidence, missing
required script inventory items, zero-length GT/reference text, too-small claim-facing
sample size, recognizer row-level claim ineligibility, or incomplete page coverage.
The direct and recipe-root `claim-safety.md` reports are also checked against their
JSON run and metric facts, using recognizer or page score fields according to
`eval_mode`, including score timestamp, machine, command provenance, hashed
checkpoint/config/dictionary artifact identity, normalization-change,
dictionary-coverage, script-inventory gaps, metadata contradictions, uncertainty
intervals, zero-length GT/reference counters, and per-language/script/domain/GT/split
slice metrics, so stale or aggregate-only reviewer-facing summaries fail audit instead
of merely having the right section headings. Direct and recipe-root score reports must
also list the review artifacts a reviewer needs to open, including `command.txt`,
canonical row artifacts, worst-example CSVs, sample-review CSV/HTML files, confusions,
and page alignment/reference artifacts.
For recipe-root audits, `recipe.json` must still match the selected recipe in the
hashed source registry and `recipe-validation.json` must still match
`recipe-run.json`; the embedded validation is also recomputed from the current recipe
registry and eval-pack audit. Recipe-run timestamp, machine, command, and root output
pointers are audited too. Root `audit.json` and the embedded recipe-run audit summary must
preserve current score-audit issues, warnings, claim-safe status, and score-dir
identity, and the recipe-run score summary must point to and hash the current
`score/metrics.json` plus the mode-specific score run file (`score/eval-run.json`
for recognizer roots, `score/page-eval-run.json` for page roots). The root train-manifest
path/hash list must also match the score eval-run leakage inputs, and the hashed
recipe registry must be under the declared recipe root. Do not hand-edit those files
after scoring.
For page-level Limbu book-candidate recipes that use
`tools/score_limbu_book_ocr_candidate.py`, the recipe validator also audits the
declared `evidence` chain. It reads the score wrapper, admission report, metrics, and
review-coverage report; verifies recorded SHA-256 values; requires champion metrics;
requires review-coverage counts for missed and extra lines; and blocks candidates with
champion regressions or uncovered review regions. It also parses the recipe
`command_template` and requires the scorer command to bind the same `--book-output`,
`--book-audit`, `--champion-metrics`, and `--review-coverage-report` paths recorded in
the evidence chain. A command string or free-text claim blocker is not enough evidence
for current-best registration, and stale evidence cannot be attached to a weaker replay
command.
The score-level `metrics.md` report is checked against `metrics.json` for core
counts, CER values, slices, reference coverage, metric means, and warnings.
Recognizer `unmatched.tsv` row identity, label image hashes, prediction scores, and
source-level match/count metrics are recomputed from
the recorded source label and prediction files, recognizer `confusions.json` is
recomputed from canonical predictions, page `confusions.json` is recomputed from
canonical `text-pairs.tsv` prediction/reference text, and page `missing.tsv` is
recomputed from the eval manifest, document outputs, and reference files.
Worst-example CSVs, reviewer CSV order and fields, and reviewer HTML samples are also
checked against the canonical scored rows, so stale or hand-edited failure samples do
not satisfy reviewability.
Completed score artifacts are checked for duplicate scored sample IDs, image paths,
document paths, and missing-row keys; duplicate IDs are not only a pre-normalization
pack failure.
For paper-claim numbers, also add `--require-claim-safe`.
## Recognizer Oracle-Crop Eval
Use this manual path for line-crop recognizers only. Do not use it for page-level
OCR claims.
### Manual Path
1. Normalize the eval pack into the canonical schema.
```bash
uv run python tools/normalize_recognizer_eval_pack.py \
--label-file data/<pack>/eval.txt \
--data-dir data/<pack> \
--manifest-file data/<pack>/manifest.jsonl \
--summary-file data/<pack>/summary.json \
--out outputs/eval-packs/<language>-<pack-version> \
--dataset <dataset-id> \
--language <language> \
--script <script> \
--domain real_print \
--gt-status machine_converted \
--split eval \
--eval-pack-version <pack-version> \
--frozen-eval-pack \
--unseen-for-selection \
--no-claim-eligible
```
If the normalizer reports missing source document, page, bbox, GT status, or claim
eligibility, do not hide it with defaults. Fix the pack or keep the result
non-claim-safe.
If a recognizer manifest row declares an image `sha256`, the scorer verifies that it
matches the current crop bytes. A stale declared hash fails scoring and fails later
audit if an older score directory points at a mutated manifest.
2. Build train/dev leakage manifests for every model family used to select or
train the checkpoint.
```bash
uv run python tools/build_recognizer_leakage_manifest.py \
--label data/<train-pack>/train.txt::data/<train-pack>::train \
--label data/<train-pack>/eval.txt::data/<train-pack>::dev \
--out outputs/leakage/<model-family>-train-dev.jsonl \
--summary outputs/leakage/<model-family>-train-dev.summary.json \
--source <model-family> \
--language <language> \
--script <script> \
--domain real_print \
--gt-status machine_converted
```
Use `--text-only` only when the corpus intentionally has no images. A text-only
manifest can catch text leakage, but it cannot catch duplicate crop images.
If crops were cut from a page eval candidate, the leakage manifest must preserve the
origin page as `source_sample_id`; otherwise a line-crop pack can look clean under exact
image/text hash checks while still being selected from the eval page.
3. Materialize predictions through a recorded command.
For repo candidate model cards, omit `--command` and provide `--model-config`:
```bash
uv run python tools/run_eval_output_command.py \
--mode recognizer \
--model-config outputs/<model-package>/model-card.json \
--out outputs/evals/<run-id>/materialize \
--system-name <system-name> \
--model-name <model-name> \
--checkpoint <local-checkpoint-file> \
--config <local-training-or-inference-config-file> \
--dictionary <local-dictionary-file> \
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
--data-dir outputs/eval-packs/<language>-<pack-version> \
--prediction-file outputs/evals/<run-id>/predictions.txt \
--timeout-seconds 3600
```
For external PaddleOCR/HF runners, pass `--command` explicitly. The command must
write `{prediction_file}` in `image<TAB>prediction<TAB>confidence` format and may
use only `{label_file}`, `{data_dir}`, `{prediction_file}`, and `{out_dir}`.
```bash
uv run python tools/run_eval_output_command.py \
--mode recognizer \
--command 'bash scripts/predict_external_recognizer.sh --labels {label_file} --data {data_dir} --out {prediction_file}' \
--out outputs/evals/<run-id>/materialize \
--system-name <system-name> \
--model-name <model-name> \
--checkpoint <local-checkpoint-file> \
--config <local-training-or-inference-config-file> \
--dictionary <local-dictionary-file> \
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
--data-dir outputs/eval-packs/<language>-<pack-version> \
--prediction-file outputs/evals/<run-id>/predictions.txt \
--timeout-seconds 3600
```
4. Score the predictions.
```bash
uv run python tools/run_recognizer_eval.py \
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
--data-dir outputs/eval-packs/<language>-<pack-version> \
--prediction-file outputs/evals/<run-id>/predictions.txt \
--out outputs/evals/<run-id>/score \
--manifest-file outputs/eval-packs/<language>-<pack-version>/manifests/recognizer-eval.jsonl \
--summary-file outputs/eval-packs/<language>-<pack-version>/summary.json \
--script-inventory-file outputs/eval-packs/<language>-<pack-version>/script-inventory.json \
--train-manifest outputs/leakage/<model-family>-train-dev.jsonl \
--model-name <model-name> \
--checkpoint <local-checkpoint-file> \
--config <local-training-or-inference-config-file> \
--dictionary <local-dictionary-file> \
--language <language> \
--script <script> \
--domain real_print \
--gt-status machine_converted \
--split eval \
--eval-pack-version <pack-version> \
--frozen-eval-pack \
--unseen-for-selection \
--no-selected-checkpoint-on-test \
--no-claim-eligible \
--min-match-rate 0.99
```
Only add `--require-claim-safe` when every warning is expected to be absent. If
the GT status is `machine_converted_spot_checked`, also pass
`--spotcheck-calibration-file`. Recipe runs can declare the same artifact under
`eval_pack.spotcheck_calibration_file` so the calibration evidence is part of the
versioned recipe instead of an invisible operator-only flag.
For a claim-safe recognizer run, `--checkpoint`, `--config`, and `--dictionary`
must point to readable local files. The scorer records their SHA-256 hashes in
`eval-run.json`, and `tools/audit_eval_run.py` re-verifies those hashes and checks
that the visible run fields still match the hashed artifact identities.
The scorer also records whether all non-whitespace GT codepoints are present in the
dictionary; missing GT codepoints are a claim-safety warning and are independently
recomputed during audit.
The same audit checks hash-valid summary/datasheet metadata against the direct run's
recorded language/script/domain/GT/split, version, frozen/unseen, default
claim-eligibility, and task metadata.
`--script-inventory-file` must also point to a readable local JSON file; it is hashed in
`eval-run.json`, summarized in `metrics.json`, and audited against the emitted GT rows.
After scoring, audit the score directory:
```bash
uv run python tools/audit_eval_run.py \
--run-dir outputs/evals/<run-id>/score \
--out outputs/evals/<run-id>/score/audit.json \
--require-audit-pass
```
### Same-Pack Recognizer Comparisons
When a paper claim compares two recognizers, do not compare separate aggregate numbers
by eye. Score both systems on the same eval pack, audit both score directories, then
run the paired comparison tool:
```bash
uv run python tools/compare_recognizer_eval_runs.py \
--candidate-score-dir outputs/evals/<candidate-run>/score \
--baseline-score-dir outputs/evals/<baseline-run>/score \
--candidate-name <candidate-name> \
--baseline-name <baseline-name> \
--out outputs/evals/<comparison-run> \
--min-pairs 30 \
--require-claim-safe
```
The tool pairs rows by `sample_id`, checks that eval-pack identity including
`pack_content_sha256` and row GT identity match, and emits `comparison.json`,
`comparison.md`, `paired-rows.tsv`, and
`command.txt`. It reports codepoint and grapheme micro-CER deltas, paired
wins/losses/ties, deterministic paired-row bootstrap confidence intervals for the
improvement, and a two-sided exact sign-test p-value. A comparison is claim-safe only
when both input score directories are claim-safe, the rows are fully paired on the same
eval pack, the bootstrap lower bound is above zero, the sign test passes the configured
alpha, the candidate actually improves the baseline, and comparison timestamp/machine/
command provenance is present and internally consistent.
Audit the comparison directory before using it in a claim:
```bash
uv run python tools/compare_recognizer_eval_runs.py \
--audit-only \
--comparison-dir outputs/evals/<comparison-run> \
--require-claim-safe
```
For page-level OCR system comparisons, use the page comparison tool instead of the
recognizer comparison tool:
```bash
uv run python tools/compare_page_ocr_eval_runs.py \
--candidate-score-dir outputs/evals/<candidate-page-run> \
--baseline-score-dir outputs/evals/<baseline-page-run> \
--candidate-name <candidate-name> \
--baseline-name <baseline-name> \
--out outputs/evals/<page-comparison-run> \
--min-pairs 30 \
--require-claim-safe
```
It pairs pages by `sample_id`, checks page eval-pack identity including
`pack_content_sha256` plus page/reference text identity, and emits `comparison.json`,
`comparison.md`, `paired-pages.tsv`, and `command.txt` for page text
codepoint/grapheme CER deltas, paired wins/losses/ties, paired-page bootstrap
intervals, and a two-sided exact sign-test p-value. Audit it the same way:
```bash
uv run python tools/compare_page_ocr_eval_runs.py \
--audit-only \
--comparison-dir outputs/evals/<page-comparison-run> \
--require-claim-safe
```
## Native-Reader Spot Check
Machine-converted labels may be scored, but they are not human-verified evidence.
Use this path to create auditable calibration evidence.
```bash
uv run python tools/prepare_native_spotcheck_bundle.py \
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
--data-dir outputs/eval-packs/<language>-<pack-version> \
--manifest-file outputs/eval-packs/<language>-<pack-version>/manifests/recognizer-eval.jsonl \
--prediction-file outputs/evals/<run-id>/predictions.txt \
--out outputs/spotchecks/<language>-<pack-version> \
--sample-size 50 \
--seed 20260619 \
--title '<language> native-reader spot check'
```
After a reviewer exports the completed CSV:
```bash
uv run python tools/apply_native_spotcheck_reviews.py \
--review-file outputs/spotchecks/<language>-<pack-version>/reviewed-spotcheck.csv \
--out outputs/spotchecks/<language>-<pack-version>/calibration \
--language <language> \
--script <script> \
--domain real_print \
--source-gt-status machine_converted \
--min-usable-rows 30 \
--max-machine-grapheme-cer 0.02 \
--max-unusable-rate 0.05
```
If the calibration output says `claim_evidence_eligible=false`, do not mark the
pack `machine_converted_spot_checked`. Completed rows without reviewer identity
are not eligible calibration evidence; fix the review export rather than
overriding the gate. The scorer also requires the calibration artifact's language,
script, domain, `source_gt_status=machine_converted`, and
`recommended_pack_gt_status=machine_converted_spot_checked` to match the scored run.
## Page-Level OCR Eval
Use this path for end-to-end page claims. Recognizer-only scores cannot support a
page OCR claim because detection, crop lineage, and reading order are absent.
1. Materialize page outputs through a recorded command.
```bash
uv run python tools/run_eval_output_command.py \
--mode page \
--model-config outputs/<page-model-package>/model-card.json \
--out outputs/page-evals/<run-id>/materialize \
--system-name <system-name> \
--model-name <model-name> \
--checkpoint <local-checkpoint-file> \
--config <local-training-or-inference-config-file> \
--dictionary <local-dictionary-file> \
--eval-manifest data/<page-eval-pack>/manifests/eval.jsonl \
--documents-dir outputs/page-evals/<run-id>/documents \
--timeout-seconds 3600
```
For an external runner, pass `--command` and write `{document_json}` for each
input page. Page commands may use only `{input_path}`, `{sample_id}`, `{output_dir}`,
`{document_json}`, `{documents_dir}`, and `{out_dir}`.
2. Score saved `document.json` outputs.
```bash
uv run python tools/run_page_ocr_eval.py \
--eval-manifest data/<page-eval-pack>/manifests/eval.jsonl \
--documents-dir outputs/page-evals/<run-id>/documents \
--ocr-output-kind live_detector \
--out outputs/page-evals/<run-id>/score \
--summary-file data/<page-eval-pack>/summary.json \
--script-inventory-file data/<page-eval-pack>/script-inventory.json \
--model-name <model-name> \
--checkpoint <local-checkpoint-file> \
--config <local-training-or-inference-config-file> \
--dictionary <local-dictionary-file> \
--eval-pack-version <pack-version> \
--frozen-eval-pack \
--unseen-for-selection \
--no-selected-checkpoint-on-test \
--train-manifest outputs/leakage/<model-family>-train-dev.jsonl
```
For direct live-detector page scoring, claim-safe mode requires a passing output audit
for every scored page output directory. By default the scorer looks for
`limbu-output-audit.json` with `passed=true` beside each `document.json`; use
`--require-output-audit`, `--output-audit-filename`, and
`--output-audit-passed-field` when scoring a different page-output audit format.
The page auditor replays those per-page audit artifacts from `page-metrics.csv` and
fails if a recorded `page_output_audit` summary no longer matches the current files,
so deleting or editing an output audit after scoring invalidates the score directory.
A paper-safe page claim requires reference text plus page/line layout references
that let the scorer report detection/alignment/reading-order behavior. If line
bboxes are absent, the result is not a full page-level OCR claim. Each page
manifest row must also carry explicit `language`, `script`, `domain`, and
`gt_status` metadata plus source-document, page, bbox, and claim-eligibility
provenance; CLI defaults are recorded as fallbacks, not as row-level claim evidence.
The page summary must carry a `pack_content_sha256` over the canonical page row,
input-image, reference-artifact, reference-text, and provenance identity; claim-safe
scoring rejects summaries whose digest is missing or stale.
Page manifest `sample_id` and `image_path` values must be unique,
and any declared row `sha256` must match the actual input image bytes; duplicate IDs,
duplicate input images, or stale image hashes fail before scoring. If page GT status
is `machine_converted_spot_checked`, pass
`--spotcheck-calibration-file`; without an eligible calibration artifact the
page result is not claim-safe.
For a claim-safe page run, `--checkpoint`, `--config`, and `--dictionary` must point
to readable local files whose SHA-256 hashes can be recorded and audited.
`--script-inventory-file` must point to a readable local JSON file; the scorer records
which required codepoints/graphemes were covered by reference text and the auditor
recomputes that summary from the eval manifest and references.
`--ocr-output-kind` must identify what kind of document outputs were scored. Use
`live_detector` for normal end-to-end page OCR outputs, `oracle_line_boxes` for
diagnostic runs where reference/PDF line boxes are fed to recognizers, and
`sidecar_fixture` only for tests or synthetic fixtures. For Limbu, the detector
diagnostic requires scoring the same frozen page pack once as `live_detector` and once
as `oracle_line_boxes`, then comparing the two score directories with
`tools/compare_page_ocr_eval_runs.py`.
Audit the page score directory:
```bash
uv run python tools/audit_eval_run.py \
--run-dir outputs/page-evals/<run-id>/score \
--out outputs/page-evals/<run-id>/score/audit.json \
--require-audit-pass
```
## Claim-Safe Gates
A recognizer or page result is not paper-safe unless all of these are true:
- Eval pack is frozen, versioned, and declared unseen for model/checkpoint
selection.
- The score records a manifest and summary/datasheet for the eval pack.
- The score records a script inventory file, its hash, observed codepoint/grapheme
coverage, and no missing required codepoints or graphemes.
- Manifest row IDs and input image paths are unique, recognizer manifest image
paths/basenames cannot collide with basename-based prediction matching, and declared
source-image hashes match the current files; stale recognizer crop hashes, duplicate
page manifest IDs or image paths, stale page manifest image hashes, and duplicate
completed score artifact IDs fail audit.
- Every scored row has source-document, page, bbox, language, script, domain,
split, GT status, and claim eligibility metadata, and every scored row is claim
eligible.
- GT status is `human_verified` or backed by eligible native-reader calibration
as `machine_converted_spot_checked`.
- Rows declared `human_verified` carry human review evidence such as `reviewed_by`,
`review_artifact`, double-review, agreement, or correction-rate metadata; the trust
label alone is not claim evidence.
- The domain is real target material, not synthetic or transliterated evidence.
- Row/reference provenance does not contradict the declared domain or GT status:
synthetic/transliterated provenance cannot be claimed as real material, and
machine-converted provenance cannot be claimed as `human_verified`.
- Train/dev leakage manifests were passed and checked.
- Model artifacts are hash-verifiable: recognizer and page runs require checkpoint,
config, and dictionary files, and
the run fields must match the hashed artifact identity entries.
- Model identity is recorded: recognizer and page runs require a non-empty model name,
and page runs require a non-empty system name.
- No hard leakage exists by sample ID, image hash, image path, or combined
sample hash.
- The prediction match rate is at least the configured threshold.
- No scored recognizer GT row or page text reference is zero-length.
- Headline reporting is per language/script/domain/GT status, not a pooled
average hiding weak languages, and every emitted language/script/domain/GT/split
slice has at least the claim-facing minimum scoring units.
- Same-pack recognizer comparisons use `tools/compare_recognizer_eval_runs.py` and pass
its paired bootstrap/sign-test gates; separate aggregate score directories are not
enough evidence for a comparative claim.
- The score directory contains `claim-safety.md` with `What Was Scored`, `Ground
Truth And Pack`, `Metrics`, and `Warnings` sections.