Download ocr-tech/docs/EVAL_HARNESS_RUNBOOK.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 44.4 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/EVAL_HARNESS_RUNBOOK.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/EVAL_HARNESS_RUNBOOK.md
-
curl -L -o EVAL_HARNESS_RUNBOOK.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/EVAL_HARNESS_RUNBOOK.md
Eval Harness Runbook
Updated: 2026-06-20
This is the operational path for producing a claim-grade OCR eval directory. The goal is not to get a number quickly; it is to make the number hard to misread. Each run should leave enough artifacts for a reviewer to answer what was scored, which model produced it, which ground truth was used, whether the pack was frozen and unseen for selection, what leakage checks ran, and why the result is or is not safe for a paper claim.
Recipe-First Eval
Use this path when a registered recipe exists for either recognizer_oracle_crop
or page_level_ocr. Recipes keep validation, leakage checks, output
materialization, scoring, root reports, and audit metadata under one output
directory.
Registry Validation
When a recipe exists in configs/eval-harness-recipes-v1.json, use it as the
default entry point. This keeps validation, leakage checks, prediction
materialization, and scoring under one output directory.
Before scoring legacy source packs, validate or run the normalization registry:
uv run python tools/run_recognizer_pack_normalization.py \
--config configs/recognizer-eval-pack-normalization-v1.json \
--root . \
--all \
--out outputs/eval-packs/normalization-runs/<run-id> \
--dry-run \
--require-valid
Drop --dry-run only when you intentionally want to materialize canonical packs
under outputs/eval-packs/<pack-version>/. A materialized pack gets its own
audit/recognizer-eval-pack-audit.json and .md; read those before wiring the
pack into a claim run.
Then validate intended language/script coverage against the normalization and eval-run recipe registries:
uv run python tools/validate_eval_target_coverage.py \
--coverage-config configs/eval-target-coverage-v1.json \
--normalization-config configs/recognizer-eval-pack-normalization-v1.json \
--eval-recipes configs/eval-harness-recipes-v1.json \
--root . \
--out outputs/evals/target-coverage.json \
--require-valid
Do not add --require-coverage-ready until every target in the selected claim
scope has a normalized eval pack, runnable eval recipe, and no claim blockers.
If a target points at a shared aggregate pack, the validator must show matching
target language/script metadata or an explicit gap; do not use a shared pack
average as evidence for an individual language claim. Coverage target status is
also gated: eval_status must be one of the known states, and only
registered_real_pack is allowed to become coverage-ready. Use
partial_real_recognizer_pack, shared_devanagari_pack, or
missing_real_eval_pack when a target is deliberately not ready for claims.
For book OCR targets, set requires_page_eval_recipe and
requires_book_candidate_evidence on the target set or target. That makes coverage
depend on a page-level recipe with book-candidate evidence from the scoring/admission
lane, not just an oracle-crop recognizer score or a loose page replay. The validator
also reports whether the linked book candidate is operationally admitted and
claim-safe, so non-claim-safe Sirijonga and Devanagari book evidence show up as
separate gaps.
uv run python tools/validate_eval_harness_recipes.py \
--recipes configs/eval-harness-recipes-v1.json \
--root . \
--require-valid
Each recipe eval_pack must declare normalization_profile: NFC_preserve_whitespace, matching the scorer/auditor recomputation contract, and a
readable, parseable summary_file and script_inventory_file. The recipe validator
checks the summary JSON object and inventory JSON shape before a recipe is considered
structurally valid, and a summary without dataset/version/eval-pack metadata blocks
claim readiness. Summary/datasheet declarations for language, script, dataset, task,
domain, GT status, split, pack version, frozen/unseen status, and default claim
eligibility must agree with the recipe metadata; contradictions block claim readiness.
Do not paper over a pack that needs another normalization policy; add
scorer support and audit coverage first, then change the recipe.
The script inventory is a JSON object with required_codepoints and
required_graphemes lists. Codepoints may be written as U+XXXX values or literal
single characters; graphemes are literal strings. The scorer records observed coverage
from GT/reference text, and claim-safe runs fail if any required item is missing.
When a runner dictionary is a readable local file, the recipe validator also compares
its non-whitespace codepoints to recognizer GT or page reference text; out-of-vocabulary
GT/reference codepoints block claim readiness before scoring.
If the recipe default or audited row metadata uses machine_converted_spot_checked
GT, the recipe must also declare a readable JSON eval_pack.spotcheck_calibration_file;
the recipe validator checks its eligibility/recommendation/source-status fields and
language/script/domain consistency, and recipe execution passes that artifact into the
scorer, where it is hashed and audited.
Pack Content Digest Contract
Every claim-facing eval pack summary/datasheet must include pack_content_sha256.
This is not a cosmetic version string. It is the immutable content identity that lets
scoring, audit, and comparison prove that the score still belongs to the same source
pack.
For recognizer oracle-crop packs, the digest binds each canonical row's sample ID, source image hash, NFC-normalized GT text hash, language, script, domain, GT status, split, and claim eligibility. For page-level OCR packs, the digest binds each canonical page row's sample ID, input image hash, reference artifact hash, reference text hash, reference fields, reference trust metadata, source-document/page/bbox provenance, language, script, domain, GT status, split, and claim eligibility.
Claim-safe scoring must fail when the summary digest is missing or stale. Completed
score directories must record the recomputed value in metrics.json and in the
mode-specific run file (eval-run.json for recognizer, page-eval-run.json for page
OCR). Audit must recompute the digest from current source artifacts and reject any
mismatch in the run JSON, metrics JSON, or summary/datasheet. Same-task comparison
tools must treat the digest as eval-pack identity, so a candidate and baseline cannot
be compared as claim-safe if their pack_content_sha256 values differ.
When a test or fixture intentionally edits pack content before scoring, regenerate the summary digest before running the scorer. When content changes after scoring, do not refresh the digest; that is tampering or drift, and audit should fail.
For page_level_ocr recipes, eval_pack.eval_manifest is the canonical page
manifest and materialization writes per-page document.json files under the root
documents/ directory. External page command templates must write either
{document_json} directly or write document.json below {output_dir}. The recipe
validator statically audits page manifests for readable source images and hashes,
source-document/page/bbox provenance, row claim eligibility, human-review evidence for
human_verified rows, parseable references, claim-ready reference metadata, and
minimum scoring-unit counts overall and per language/script/domain/GT/split slice; any
page audit issue or warning blocks claim readiness. For path-complete recognizer packs,
the recipe validator also runs
tools/audit_recognizer_eval_pack.py; any pack-audit issue or warning becomes a
claim-readiness blocker, including overall and language/script/domain/GT/split
sample-size warnings below the claim minimum. leakage_sources must also be
parseable and non-empty; page-level leakage manifests must satisfy the page manifest
schema, and recognizer label leakage sources must contain valid rows with existing
images. A recipe is not claim-ready merely because its label, manifest, summary,
script inventory, and image paths exist.
For Limbu, keep the pack counts straight. The canonical materialized recognizer pack
outputs/eval-packs/limbu-gorkhapatra-real-v1/ has 1,829 labels and 1,829 manifest
rows, not 39. The number 39 belongs to an older operational page run's unresolved
review queue and must not be described as the available Limbu eval set. The current
Limbu recognizer pack is structurally auditable but not claim-ready: it is
machine-converted, has 0 claim-eligible rows, is not declared unseen for selection, and
is missing source-document/page/bbox provenance. The separate CDC Maths PDF text-layer
pack has 81 silver rendered-line rows, split 69 train / 12 eval; use it for recognizer
bootstrap/training experiments, not for claim-safe evaluation.
For page_level_ocr recipes with ocr_output_kind=live_detector, the recipe must
also declare runner.output_audit bound to {output_dir} so page scoring is paired
with an operational artifact audit. Limbu page recipes must run audit-limbu-output
and record strict flags including --require-capture-prep, prepared-image quality
thresholds (--min-capture-prepared-width, --min-capture-prepared-height,
--min-capture-prepared-entropy, --min-capture-prepared-luminance-stddev, and
--min-capture-prepared-edge-stddev), --require-no-pending-review,
--require-no-dropped-image-lines, output quality/volume gates
(--min-line-count, --min-average-line-confidence, and --min-quality-score), and
the script-specific --require-script value for their target, for example
limbu_sirijonga for Sirijonga-script Limbu or devanagari_limbu for
Devanagari-written Limbu. This keeps page scoring tied to the operational artifact
audit instead of allowing a claim-ready registry entry for an OCR bundle that would
fail provenance, capture-quality, output-volume, output-quality, review-resolution, or
complete-line-retention gates.
Then run the selected recipe:
uv run python tools/run_eval_recipe.py \
--recipes configs/eval-harness-recipes-v1.json \
--recipe-id <recipe-id> \
--root . \
--out outputs/evals/<run-id> \
--timeout-seconds 3600 \
--min-match-rate 0.99
Add --require-claim-safe only when the recipe validator reports no claim
blockers and the scorer is expected to emit no warnings. If the recipe uses an
external command template, replace the placeholder command in the recipe first;
unresolved templates are rejected by default. Recognizer external templates must include
{label_file}, {data_dir}, and {prediction_file}; page external templates must
include {input_path} plus {document_json} or {output_dir}. Unknown placeholders
are validation errors, so command templates must use only the mode-supported
placeholders documented by tools/run_eval_output_command.py. Recipes that declare
runner.selected_checkpoint_on_test or non-empty known_leakage remain structurally
valid but are blocked from claim readiness because the scorer will warn on those same
facts at runtime. Runner checkpoint, config, and dictionary values must point to
readable local files, not directories, because claim-safe scores record and audit their
SHA-256 hashes.
Recognizer and page eval-pack summaries must also carry pack_content_sha256, the
canonical digest over sample ID, input/reference hashes, normalized GT text or
structured-reference text, provenance, language, script, domain, GT status, split, and
claim eligibility. The scorers recompute this digest from the rows they actually
scored; claim-safe runs fail when the summary digest is missing or stale.
The recipe runner writes the root-level files a reviewer should open first:
recipe-run.json, audit.json, and claim-safety.md. For page-level recipes with
runner.output_audit, execution runs that audit once per materialized page output
directory before scoring and records output-audit/output-audit-run.json plus
output-audit/command-template.txt and Markdown with commands, logs, return codes,
and available audit artifact hashes. The runner honors
runner.output_audit.result_filename when recording the per-page audit JSON; if it is
omitted, the default is limbu-output-audit.json. The root audit requires each
recorded per-page audit JSON to still exist with the recorded SHA-256. It also
requires the output-audit command template sidecar when recipe-run.json records an
output-audit command, rejects a missing or stale sidecar, and replays each per-page
output-audit command from that template using the recorded output directory. It
also replays the recorded stdout/stderr log SHA-256 values for each per-page audit
command. It requires output-audit/output-audit-run.md and checks that the Markdown
report still reflects the recorded count, per-page output audit rows, log hashes, and
recorded audit artifact hashes.
The report summarizes what was scored, the GT trust
metadata, frozen/unseen declarations, leakage status, micro-CER metrics, output-audit
status, warnings, and review artifact locations. The root report is audited against
recipe-run.json, the score eval-run JSON, and metrics.json, with mode-specific
expectations for recognizer oracle-crop and page-level OCR score facts, so stale
top-level summaries fail audit.
The recipe-root audit also checks that materialize/output-command-run.json matches
the materialization summary embedded in recipe-run.json, that
materialize/command-template.txt and materialize/output-command-run.md match the
recorded command/output summary, that materialization timestamp, machine platform/
Python metadata, invocation, per-record command, and stdout/stderr log paths and
SHA-256 values are present and replay against current log bytes. It also replays
materialization record commands from the recorded command template using the recorded
recognizer or page materialization fields; if those fields are missing, the root audit
fails instead of skipping command replay. For recognizer roots, the root predictions.txt path and hash must be the
same prediction artifact recorded by the score eval-run JSON. For page roots,
each materialization record must carry the source page image path/hash, the recorded
hash is rechecked against current bytes, outputs.documents_dir must match the score
eval-run documents_dir, and materialized document.json outputs must live under that directory. Recipe-root materialization
model/checkpoint/config/dictionary metadata is also checked against the score eval-run
metadata.
If materialization fails before a score directory exists, keep
materialize/output-command-run-error.json: it carries the timestamp, machine,
invocation, model/checkpoint/config/dictionary metadata, command template, and label
or eval-manifest path/hash used for the failed attempt. If the recipe wrapper fails,
keep recipe-run-error.json: it records the timestamp, machine, command, recipe
registry hash, selected recipe validation, root, output directory, exception type,
message, and traceback.
The underlying recognizer or page score directory also writes its own
claim-safety.md, so direct scorer runs are reviewable without a recipe wrapper.
Audit the finished run directory before reporting the number:
uv run python tools/audit_eval_run.py \
--run-dir outputs/evals/<run-id> \
--out outputs/evals/<run-id>/audit.json \
--require-audit-pass
The audit recomputes recognizer row edits and aggregate micro-CER from
score/predictions.tsv; a metric file that is internally ratio-consistent but does
not match the emitted GT/prediction rows must fail audit. The run machine metadata
must include concrete platform and python values, and normalization_profile
must match the scorer-supported NFC_preserve_whitespace profile used by the audit
recomputation path. The audit also recomputes normalization_summary from
predictions.tsv for recognizer runs and text-pairs.tsv for page runs; those
fields count raw GT/reference and prediction strings changed by NFC, so
normalization-heavy scores remain visible instead of being hidden behind normalized
CER. The audit
also reconstructs predictions.tsv row identity, GT text, prediction text,
prediction score, crop hash, language/script/domain/GT/split, and row
claim-eligibility fields from the recorded label, prediction, and manifest sources.
The same file carries
language/script/domain/GT/split, claim-eligibility, and crop image SHA-256 values, so
recognizer slice metrics, claim-eligibility provenance, and crop-image identity are
recomputed from the emitted row artifact, not trusted as separate aggregates.
worst-lines.csv is also audited against those canonical prediction rows, including
crop hashes, trust metadata, claim eligibility, and source-document/page/bbox
provenance, so a failure sample cannot be made more claim-friendly by editing the
review artifact after scoring.
Recognizer label-line count, scored-line count, matched-line count, unmatched label
count, unmatched prediction count, and prediction match rate are recomputed from the
recorded source label and prediction files.
Recognizer normalizer_counts in metrics.json and eval-run.json are also checked
against the recorded summary/datasheet counts, so defaulted pack metadata cannot be
hidden by editing both score JSON files after scoring.
Recognizer and page summary/datasheet files are semantically audited too: if a
hash-valid summary declares language, script, domain, GT status, split, eval-pack
version, frozen/unseen status, default claim eligibility, or task metadata that
contradicts the scored run metadata, the audit fails.
Run-level language/script/domain/GT/split lists in eval-run.json are also checked
against the unique values in predictions.tsv.
predictions.tsv also carries source-document/page/bbox provenance columns, and the
audit reconstructs those values from the recorded label/manifest sources.
For page runs, the audit also recomputes line-alignment.csv row contents from the
scored document.json and reference JSON, including predicted/reference line IDs,
line bboxes, text, IoU, and per-line text CER, then recomputes line detection counts,
precision/recall/F1, mean IoU, and aggregate metric means from line-alignment.csv
and page-metrics.csv.
The audit also reconstructs page-metrics.csv row identity from the eval manifest,
resolved document.json, embedded document.source_path, scorer metadata defaults,
and parsed reference fields, so
language/script/domain/GT/split, source-document/page/bbox provenance,
claim_eligible, and reference_fields cannot be changed after scoring without
failing audit.
Run-level language/script/domain/GT/split lists in page-eval-run.json must match
the unique values in page-metrics.csv.
reference-artifacts.tsv carries the exact reference path, SHA-256, scoreable
reference fields, text hash, and reference trust metadata for each scored page, and
the audit recomputes it from the eval manifest and reference files.
worst-pages.csv is audited against page-metrics.csv plus
reference-artifacts.tsv, including input/document hashes, language/script/domain/GT
metadata, claim eligibility, source-document/page/bbox provenance, reference hashes,
and reference trust fields.
page-metrics.csv carries raw page text edit/ref counts and exact text-match flags,
text-pairs.tsv carries the exact prediction/reference page text used for those
counts, and the audit recomputes the text-pair artifact from scored document.json
text plus reference text before rolling it into page text micro-CER,
text exact-accuracy totals, and per-language/script/domain/GT/split page slices.
It also records source page image and document.json SHA-256 values, which the audit
rechecks so a page score cannot point at mutated OCR outputs after the fact.
Train/dev/eval leakage reports are recomputed from the recorded train manifests and
canonical eval rows/manifests, including checked state, row counts, and overlap sets.
For page-derived crop training or checkpoint-selection data, include source_sample_id
on train rows; scorer and audit treat overlap between train source_sample_id and eval
page sample_id as hard leakage even when crop image hashes and line text hashes do not
match the full page row directly.
Native spot-check calibration summaries embedded in run and metric JSON are checked
against the recorded calibration artifact.
Metadata contradictions are recomputed from manifest/reference provenance signals and
canonical row domain/GT fields.
Recognizer provenance coverage and page row metadata coverage counts are recomputed
from their source manifests, and page reference coverage is recomputed from
page-metrics.csv. Page manifest/scored/missing/claim-eligible counts are recomputed
from the eval manifest, document outputs, references, and claim-eligibility metadata.
Recognizer and page score directories also carry deterministic bootstrap uncertainty
summaries in metrics.json; the audit recomputes those intervals from
predictions.tsv or page-metrics.csv. Claim-safe runs currently require at least
30 scored rows/pages, so one-page smoke checks and tiny oracle-crop packs must remain
non-claim-safe even when their point CER is perfect.
Recognizer and page score directories also carry script inventory summaries in
metrics.json and run JSON; the audit recomputes observed codepoints/graphemes from
recognizer GT rows or page reference text and verifies the inventory file hash. A
missing inventory file, a stale inventory summary, or missing required
codepoints/graphemes remains a claim-safety blocker even when CER is low.
They also carry dictionary_coverage in metrics.json; the audit rehashes the
recorded dictionary file, recomputes the non-whitespace GT/reference codepoints, and
blocks claim-safe status when the ground truth contains codepoints outside the
dictionary vocabulary. metrics.md and claim-safety.md must surface the same
dictionary coverage, normalization-change, and zero-length GT/reference counters,
and the audit treats stale Markdown as an output-directory integrity failure.
Recognizer metrics record zero_length_gt_lines, and page metrics record
zero_length_text_references; the audit recomputes both from canonical rows and
blocks claim-safe status when any scored GT/reference text is empty.
The audit also derives claim-safety blockers from durable run facts, so a tampered
claim_safe=true field without matching warnings still fails if the artifacts show
metadata contradictions, a missing manifest or summary/datasheet, an unrun leakage
check, leakage overlaps, low recognizer match rate, unsafe GT/domain status, missing
provenance, missing uncertainty evidence, missing script inventory evidence, missing
required script inventory items, zero-length GT/reference text, too-small claim-facing
sample size, recognizer row-level claim ineligibility, or incomplete page coverage.
The direct and recipe-root claim-safety.md reports are also checked against their
JSON run and metric facts, using recognizer or page score fields according to
eval_mode, including score timestamp, machine, command provenance, hashed
checkpoint/config/dictionary artifact identity, normalization-change,
dictionary-coverage, script-inventory gaps, metadata contradictions, uncertainty
intervals, zero-length GT/reference counters, and per-language/script/domain/GT/split
slice metrics, so stale or aggregate-only reviewer-facing summaries fail audit instead
of merely having the right section headings. Direct and recipe-root score reports must
also list the review artifacts a reviewer needs to open, including command.txt,
canonical row artifacts, worst-example CSVs, sample-review CSV/HTML files, confusions,
and page alignment/reference artifacts.
For recipe-root audits, recipe.json must still match the selected recipe in the
hashed source registry and recipe-validation.json must still match
recipe-run.json; the embedded validation is also recomputed from the current recipe
registry and eval-pack audit. Recipe-run timestamp, machine, command, and root output
pointers are audited too. Root audit.json and the embedded recipe-run audit summary must
preserve current score-audit issues, warnings, claim-safe status, and score-dir
identity, and the recipe-run score summary must point to and hash the current
score/metrics.json plus the mode-specific score run file (score/eval-run.json
for recognizer roots, score/page-eval-run.json for page roots). The root train-manifest
path/hash list must also match the score eval-run leakage inputs, and the hashed
recipe registry must be under the declared recipe root. Do not hand-edit those files
after scoring.
For page-level Limbu book-candidate recipes that use
tools/score_limbu_book_ocr_candidate.py, the recipe validator also audits the
declared evidence chain. It reads the score wrapper, admission report, metrics, and
review-coverage report; verifies recorded SHA-256 values; requires champion metrics;
requires review-coverage counts for missed and extra lines; and blocks candidates with
champion regressions or uncovered review regions. It also parses the recipe
command_template and requires the scorer command to bind the same --book-output,
--book-audit, --champion-metrics, and --review-coverage-report paths recorded in
the evidence chain. A command string or free-text claim blocker is not enough evidence
for current-best registration, and stale evidence cannot be attached to a weaker replay
command.
The score-level metrics.md report is checked against metrics.json for core
counts, CER values, slices, reference coverage, metric means, and warnings.
Recognizer unmatched.tsv row identity, label image hashes, prediction scores, and
source-level match/count metrics are recomputed from
the recorded source label and prediction files, recognizer confusions.json is
recomputed from canonical predictions, page confusions.json is recomputed from
canonical text-pairs.tsv prediction/reference text, and page missing.tsv is
recomputed from the eval manifest, document outputs, and reference files.
Worst-example CSVs, reviewer CSV order and fields, and reviewer HTML samples are also
checked against the canonical scored rows, so stale or hand-edited failure samples do
not satisfy reviewability.
Completed score artifacts are checked for duplicate scored sample IDs, image paths,
document paths, and missing-row keys; duplicate IDs are not only a pre-normalization
pack failure.
For paper-claim numbers, also add --require-claim-safe.
Recognizer Oracle-Crop Eval
Use this manual path for line-crop recognizers only. Do not use it for page-level OCR claims.
Manual Path
- Normalize the eval pack into the canonical schema.
uv run python tools/normalize_recognizer_eval_pack.py \
--label-file data/<pack>/eval.txt \
--data-dir data/<pack> \
--manifest-file data/<pack>/manifest.jsonl \
--summary-file data/<pack>/summary.json \
--out outputs/eval-packs/<language>-<pack-version> \
--dataset <dataset-id> \
--language <language> \
--script <script> \
--domain real_print \
--gt-status machine_converted \
--split eval \
--eval-pack-version <pack-version> \
--frozen-eval-pack \
--unseen-for-selection \
--no-claim-eligible
If the normalizer reports missing source document, page, bbox, GT status, or claim
eligibility, do not hide it with defaults. Fix the pack or keep the result
non-claim-safe.
If a recognizer manifest row declares an image sha256, the scorer verifies that it
matches the current crop bytes. A stale declared hash fails scoring and fails later
audit if an older score directory points at a mutated manifest.
- Build train/dev leakage manifests for every model family used to select or train the checkpoint.
uv run python tools/build_recognizer_leakage_manifest.py \
--label data/<train-pack>/train.txt::data/<train-pack>::train \
--label data/<train-pack>/eval.txt::data/<train-pack>::dev \
--out outputs/leakage/<model-family>-train-dev.jsonl \
--summary outputs/leakage/<model-family>-train-dev.summary.json \
--source <model-family> \
--language <language> \
--script <script> \
--domain real_print \
--gt-status machine_converted
Use --text-only only when the corpus intentionally has no images. A text-only
manifest can catch text leakage, but it cannot catch duplicate crop images.
If crops were cut from a page eval candidate, the leakage manifest must preserve the
origin page as source_sample_id; otherwise a line-crop pack can look clean under exact
image/text hash checks while still being selected from the eval page.
- Materialize predictions through a recorded command.
For repo candidate model cards, omit --command and provide --model-config:
uv run python tools/run_eval_output_command.py \
--mode recognizer \
--model-config outputs/<model-package>/model-card.json \
--out outputs/evals/<run-id>/materialize \
--system-name <system-name> \
--model-name <model-name> \
--checkpoint <local-checkpoint-file> \
--config <local-training-or-inference-config-file> \
--dictionary <local-dictionary-file> \
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
--data-dir outputs/eval-packs/<language>-<pack-version> \
--prediction-file outputs/evals/<run-id>/predictions.txt \
--timeout-seconds 3600
For external PaddleOCR/HF runners, pass --command explicitly. The command must
write {prediction_file} in image<TAB>prediction<TAB>confidence format and may
use only {label_file}, {data_dir}, {prediction_file}, and {out_dir}.
uv run python tools/run_eval_output_command.py \
--mode recognizer \
--command 'bash scripts/predict_external_recognizer.sh --labels {label_file} --data {data_dir} --out {prediction_file}' \
--out outputs/evals/<run-id>/materialize \
--system-name <system-name> \
--model-name <model-name> \
--checkpoint <local-checkpoint-file> \
--config <local-training-or-inference-config-file> \
--dictionary <local-dictionary-file> \
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
--data-dir outputs/eval-packs/<language>-<pack-version> \
--prediction-file outputs/evals/<run-id>/predictions.txt \
--timeout-seconds 3600
- Score the predictions.
uv run python tools/run_recognizer_eval.py \
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
--data-dir outputs/eval-packs/<language>-<pack-version> \
--prediction-file outputs/evals/<run-id>/predictions.txt \
--out outputs/evals/<run-id>/score \
--manifest-file outputs/eval-packs/<language>-<pack-version>/manifests/recognizer-eval.jsonl \
--summary-file outputs/eval-packs/<language>-<pack-version>/summary.json \
--script-inventory-file outputs/eval-packs/<language>-<pack-version>/script-inventory.json \
--train-manifest outputs/leakage/<model-family>-train-dev.jsonl \
--model-name <model-name> \
--checkpoint <local-checkpoint-file> \
--config <local-training-or-inference-config-file> \
--dictionary <local-dictionary-file> \
--language <language> \
--script <script> \
--domain real_print \
--gt-status machine_converted \
--split eval \
--eval-pack-version <pack-version> \
--frozen-eval-pack \
--unseen-for-selection \
--no-selected-checkpoint-on-test \
--no-claim-eligible \
--min-match-rate 0.99
Only add --require-claim-safe when every warning is expected to be absent. If
the GT status is machine_converted_spot_checked, also pass
--spotcheck-calibration-file. Recipe runs can declare the same artifact under
eval_pack.spotcheck_calibration_file so the calibration evidence is part of the
versioned recipe instead of an invisible operator-only flag.
For a claim-safe recognizer run, --checkpoint, --config, and --dictionary
must point to readable local files. The scorer records their SHA-256 hashes in
eval-run.json, and tools/audit_eval_run.py re-verifies those hashes and checks
that the visible run fields still match the hashed artifact identities.
The scorer also records whether all non-whitespace GT codepoints are present in the
dictionary; missing GT codepoints are a claim-safety warning and are independently
recomputed during audit.
The same audit checks hash-valid summary/datasheet metadata against the direct run's
recorded language/script/domain/GT/split, version, frozen/unseen, default
claim-eligibility, and task metadata.
--script-inventory-file must also point to a readable local JSON file; it is hashed in
eval-run.json, summarized in metrics.json, and audited against the emitted GT rows.
After scoring, audit the score directory:
uv run python tools/audit_eval_run.py \
--run-dir outputs/evals/<run-id>/score \
--out outputs/evals/<run-id>/score/audit.json \
--require-audit-pass
Same-Pack Recognizer Comparisons
When a paper claim compares two recognizers, do not compare separate aggregate numbers by eye. Score both systems on the same eval pack, audit both score directories, then run the paired comparison tool:
uv run python tools/compare_recognizer_eval_runs.py \
--candidate-score-dir outputs/evals/<candidate-run>/score \
--baseline-score-dir outputs/evals/<baseline-run>/score \
--candidate-name <candidate-name> \
--baseline-name <baseline-name> \
--out outputs/evals/<comparison-run> \
--min-pairs 30 \
--require-claim-safe
The tool pairs rows by sample_id, checks that eval-pack identity including
pack_content_sha256 and row GT identity match, and emits comparison.json,
comparison.md, paired-rows.tsv, and
command.txt. It reports codepoint and grapheme micro-CER deltas, paired
wins/losses/ties, deterministic paired-row bootstrap confidence intervals for the
improvement, and a two-sided exact sign-test p-value. A comparison is claim-safe only
when both input score directories are claim-safe, the rows are fully paired on the same
eval pack, the bootstrap lower bound is above zero, the sign test passes the configured
alpha, the candidate actually improves the baseline, and comparison timestamp/machine/
command provenance is present and internally consistent.
Audit the comparison directory before using it in a claim:
uv run python tools/compare_recognizer_eval_runs.py \
--audit-only \
--comparison-dir outputs/evals/<comparison-run> \
--require-claim-safe
For page-level OCR system comparisons, use the page comparison tool instead of the recognizer comparison tool:
uv run python tools/compare_page_ocr_eval_runs.py \
--candidate-score-dir outputs/evals/<candidate-page-run> \
--baseline-score-dir outputs/evals/<baseline-page-run> \
--candidate-name <candidate-name> \
--baseline-name <baseline-name> \
--out outputs/evals/<page-comparison-run> \
--min-pairs 30 \
--require-claim-safe
It pairs pages by sample_id, checks page eval-pack identity including
pack_content_sha256 plus page/reference text identity, and emits comparison.json,
comparison.md, paired-pages.tsv, and command.txt for page text
codepoint/grapheme CER deltas, paired wins/losses/ties, paired-page bootstrap
intervals, and a two-sided exact sign-test p-value. Audit it the same way:
uv run python tools/compare_page_ocr_eval_runs.py \
--audit-only \
--comparison-dir outputs/evals/<page-comparison-run> \
--require-claim-safe
Native-Reader Spot Check
Machine-converted labels may be scored, but they are not human-verified evidence. Use this path to create auditable calibration evidence.
uv run python tools/prepare_native_spotcheck_bundle.py \
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
--data-dir outputs/eval-packs/<language>-<pack-version> \
--manifest-file outputs/eval-packs/<language>-<pack-version>/manifests/recognizer-eval.jsonl \
--prediction-file outputs/evals/<run-id>/predictions.txt \
--out outputs/spotchecks/<language>-<pack-version> \
--sample-size 50 \
--seed 20260619 \
--title '<language> native-reader spot check'
After a reviewer exports the completed CSV:
uv run python tools/apply_native_spotcheck_reviews.py \
--review-file outputs/spotchecks/<language>-<pack-version>/reviewed-spotcheck.csv \
--out outputs/spotchecks/<language>-<pack-version>/calibration \
--language <language> \
--script <script> \
--domain real_print \
--source-gt-status machine_converted \
--min-usable-rows 30 \
--max-machine-grapheme-cer 0.02 \
--max-unusable-rate 0.05
If the calibration output says claim_evidence_eligible=false, do not mark the
pack machine_converted_spot_checked. Completed rows without reviewer identity
are not eligible calibration evidence; fix the review export rather than
overriding the gate. The scorer also requires the calibration artifact's language,
script, domain, source_gt_status=machine_converted, and
recommended_pack_gt_status=machine_converted_spot_checked to match the scored run.
Page-Level OCR Eval
Use this path for end-to-end page claims. Recognizer-only scores cannot support a page OCR claim because detection, crop lineage, and reading order are absent.
- Materialize page outputs through a recorded command.
uv run python tools/run_eval_output_command.py \
--mode page \
--model-config outputs/<page-model-package>/model-card.json \
--out outputs/page-evals/<run-id>/materialize \
--system-name <system-name> \
--model-name <model-name> \
--checkpoint <local-checkpoint-file> \
--config <local-training-or-inference-config-file> \
--dictionary <local-dictionary-file> \
--eval-manifest data/<page-eval-pack>/manifests/eval.jsonl \
--documents-dir outputs/page-evals/<run-id>/documents \
--timeout-seconds 3600
For an external runner, pass --command and write {document_json} for each
input page. Page commands may use only {input_path}, {sample_id}, {output_dir},
{document_json}, {documents_dir}, and {out_dir}.
- Score saved
document.jsonoutputs.
uv run python tools/run_page_ocr_eval.py \
--eval-manifest data/<page-eval-pack>/manifests/eval.jsonl \
--documents-dir outputs/page-evals/<run-id>/documents \
--ocr-output-kind live_detector \
--out outputs/page-evals/<run-id>/score \
--summary-file data/<page-eval-pack>/summary.json \
--script-inventory-file data/<page-eval-pack>/script-inventory.json \
--model-name <model-name> \
--checkpoint <local-checkpoint-file> \
--config <local-training-or-inference-config-file> \
--dictionary <local-dictionary-file> \
--eval-pack-version <pack-version> \
--frozen-eval-pack \
--unseen-for-selection \
--no-selected-checkpoint-on-test \
--train-manifest outputs/leakage/<model-family>-train-dev.jsonl
For direct live-detector page scoring, claim-safe mode requires a passing output audit
for every scored page output directory. By default the scorer looks for
limbu-output-audit.json with passed=true beside each document.json; use
--require-output-audit, --output-audit-filename, and
--output-audit-passed-field when scoring a different page-output audit format.
The page auditor replays those per-page audit artifacts from page-metrics.csv and
fails if a recorded page_output_audit summary no longer matches the current files,
so deleting or editing an output audit after scoring invalidates the score directory.
A paper-safe page claim requires reference text plus page/line layout references
that let the scorer report detection/alignment/reading-order behavior. If line
bboxes are absent, the result is not a full page-level OCR claim. Each page
manifest row must also carry explicit language, script, domain, and
gt_status metadata plus source-document, page, bbox, and claim-eligibility
provenance; CLI defaults are recorded as fallbacks, not as row-level claim evidence.
The page summary must carry a pack_content_sha256 over the canonical page row,
input-image, reference-artifact, reference-text, and provenance identity; claim-safe
scoring rejects summaries whose digest is missing or stale.
Page manifest sample_id and image_path values must be unique,
and any declared row sha256 must match the actual input image bytes; duplicate IDs,
duplicate input images, or stale image hashes fail before scoring. If page GT status
is machine_converted_spot_checked, pass
--spotcheck-calibration-file; without an eligible calibration artifact the
page result is not claim-safe.
For a claim-safe page run, --checkpoint, --config, and --dictionary must point
to readable local files whose SHA-256 hashes can be recorded and audited.
--script-inventory-file must point to a readable local JSON file; the scorer records
which required codepoints/graphemes were covered by reference text and the auditor
recomputes that summary from the eval manifest and references.
--ocr-output-kind must identify what kind of document outputs were scored. Use
live_detector for normal end-to-end page OCR outputs, oracle_line_boxes for
diagnostic runs where reference/PDF line boxes are fed to recognizers, and
sidecar_fixture only for tests or synthetic fixtures. For Limbu, the detector
diagnostic requires scoring the same frozen page pack once as live_detector and once
as oracle_line_boxes, then comparing the two score directories with
tools/compare_page_ocr_eval_runs.py.
Audit the page score directory:
uv run python tools/audit_eval_run.py \
--run-dir outputs/page-evals/<run-id>/score \
--out outputs/page-evals/<run-id>/score/audit.json \
--require-audit-pass
Claim-Safe Gates
A recognizer or page result is not paper-safe unless all of these are true:
- Eval pack is frozen, versioned, and declared unseen for model/checkpoint selection.
- The score records a manifest and summary/datasheet for the eval pack.
- The score records a script inventory file, its hash, observed codepoint/grapheme coverage, and no missing required codepoints or graphemes.
- Manifest row IDs and input image paths are unique, recognizer manifest image paths/basenames cannot collide with basename-based prediction matching, and declared source-image hashes match the current files; stale recognizer crop hashes, duplicate page manifest IDs or image paths, stale page manifest image hashes, and duplicate completed score artifact IDs fail audit.
- Every scored row has source-document, page, bbox, language, script, domain, split, GT status, and claim eligibility metadata, and every scored row is claim eligible.
- GT status is
human_verifiedor backed by eligible native-reader calibration asmachine_converted_spot_checked. - Rows declared
human_verifiedcarry human review evidence such asreviewed_by,review_artifact, double-review, agreement, or correction-rate metadata; the trust label alone is not claim evidence. - The domain is real target material, not synthetic or transliterated evidence.
- Row/reference provenance does not contradict the declared domain or GT status:
synthetic/transliterated provenance cannot be claimed as real material, and
machine-converted provenance cannot be claimed as
human_verified. - Train/dev leakage manifests were passed and checked.
- Model artifacts are hash-verifiable: recognizer and page runs require checkpoint, config, and dictionary files, and the run fields must match the hashed artifact identity entries.
- Model identity is recorded: recognizer and page runs require a non-empty model name, and page runs require a non-empty system name.
- No hard leakage exists by sample ID, image hash, image path, or combined sample hash.
- The prediction match rate is at least the configured threshold.
- No scored recognizer GT row or page text reference is zero-length.
- Headline reporting is per language/script/domain/GT status, not a pooled average hiding weak languages, and every emitted language/script/domain/GT/split slice has at least the claim-facing minimum scoring units.
- Same-pack recognizer comparisons use
tools/compare_recognizer_eval_runs.pyand pass its paired bootstrap/sign-test gates; separate aggregate score directories are not enough evidence for a comparative claim. - The score directory contains
claim-safety.mdwithWhat Was Scored,Ground Truth And Pack,Metrics, andWarningssections.