# Evaluation Harness Goal Updated: 2026-06-20 ## Current Status This is a parked claim-safety design, not the active product work. The active 2026-06-23 objective is Limbu book OCR: real book pages in, editable Unicode book text out, with line structure preserved and unresolved text surfaced for human review. Do not add generic eval-harness features from this document unless they directly unblock that OCR loop. For the current Limbu work, the useful evaluation surface is deliberately thin: frozen real-book page acceptance checks, artifact audits, page/line counts, missed/extra/blank-line rates, review-packet coverage, and reviewed-only training-pack export validation. ## Goal The goal is a claim-grade OCR evaluation harness for the Nepal/diaspora recognizer suite that makes weak publication claims difficult by default, not merely inconvenient: each language/script result must be scored only from a frozen, versioned eval pack whose source images, labels, manifest rows, source-document/page/bbox provenance, split role, domain, ground-truth trust evidence, claim eligibility, canonical pack-content digest, and script/grapheme coverage are auditable; each run must record the exact model, checkpoint, config, dictionary, prediction file, normalization profile, command, timestamp, machine, pack-content digest, and artifact hashes; each metric must be recomputed from canonical predictions as corpus micro edit-distance CER over both Unicode codepoints and UAX #29 extended grapheme clusters, with exact-line accuracy, matched/unmatched rows, worst examples, confidence intervals, and paired comparison tests when claims compare systems; recognizer-only oracle-crop scores must remain separate from page-level OCR scores that include detection, crop lineage, reading order, and alignment; and the harness must fail or hard-warn on missing images, duplicate IDs, low match rate, train/dev/test leakage, checkpoint selection on test data, synthetic or transliterated data presented as real evidence, machine-converted labels presented as human-verified evidence, human GT without review/agreement evidence, eval packs too small for the claimed conclusion, uncovered script inventories, and aggregate averages that hide per-language, per-script, per-domain, or per-GT-status failures. A reviewer opening one output directory should be able to decide what was scored, against which ground truth, how trustworthy that ground truth is, whether the test set was frozen and unseen for selection, what normalization and metric were used, how uncertain the number is, what the failures look like, what script material was not covered, and whether the result is safe for a paper claim. ## Requirements - **Eval packs are versioned evidence objects.** Each pack must include `images/`, a label file, a manifest JSONL, and a summary/datasheet containing source documents, page/bbox provenance, language, script, domain, split role, GT source, GT trust level, known leakage status, claim eligibility, and a `pack_content_sha256` over the canonical row identity/text/provenance fields so pack drift is detectable across normalization, scoring, audit, and comparison. - **Metrics are canonical and micro-averaged.** The harness must compute total-edit-distance / total-reference-length CER, not per-line macro averages. It must report both codepoint CER and grapheme-cluster CER, plus exact-line accuracy, matched/unmatched counts, prediction match rate, total reference units, and worst-line examples. - **Point estimates are not enough for claims.** Every claim-facing metric must include confidence intervals computed by resampling the scoring units, and same-pack system comparisons must include a paired test over per-row errors. The report must show the effective sample size per language/script/domain and hard-warn when a pack is too small for the claimed conclusion. - **PaddleOCR training metrics are not claim metrics.** PaddleOCR `RecMetric` reports exact-line `acc` and `norm_edit_dis` as one minus the average per-sample normalized Levenshtein distance; this may be useful during training, but it is not the corpus-level micro CER used for claims. - **Recognition and page OCR are separate tasks.** Oracle-crop recognizer results are valid recognizer evidence only. Page-level OCR claims require page images, detector output, crop/reading-order/alignment logic, and a separate end-to-end score. - **Unicode handling is explicit.** Text is normalized with a named profile, defaulting to NFC with whitespace preserved. The audit must reject a stale or unsupported profile that no longer matches the metric recomputation path. Metrics must also report how many raw GT/reference and prediction strings changed under that profile, and the audit must recompute those counts from canonical row artifacts so hidden normalization effects cannot disappear into a single CER. Grapheme metrics use Unicode extended grapheme clusters; any script-specific tailoring must be recorded. - **Human verification is first-class metadata.** Machine-converted GT may be scored, but the output must label it as machine-converted. Paper-grade human-reading claims require either fully reviewed GT or a documented native-reader calibration sample. Human-verified GT must record who reviewed it, how much was double-reviewed, and the observed disagreement or correction rate; a trust label without review evidence is not claim evidence. - **Script coverage is evidence, not decoration.** Each eval pack must report the codepoint, grapheme-cluster, digit, punctuation, conjunct/combining-mark, and language-specific inventory coverage it exercises, plus gaps against the intended script inventory. A low CER on a pack that omits common graphemes or conjuncts must not be eligible for broad script support claims. - **Reports are inspectable.** Each run should emit `eval-run.json`, `metrics.json`, `metrics.md`, `predictions.tsv`, source-audited `unmatched.tsv`, `worst-lines.csv`, `worst-pages.csv`, `sample-review.csv`, `sample-review.html`, `confusions.json`, `command.txt`, and a human-readable `claim-safety.md` report. Worst-example rows must carry the row identity, hashes, language/script/domain/GT/split, claim eligibility, source-document/page/bbox provenance, and reference-artifact trust fields needed to inspect a failure without reverse-engineering the rest of the run. - **Reporting is per language/script/domain.** Shared recognizers, such as the Devanagari model, must still report Majhi, Magar, Sunuwar-Devanagari, Magahi, Bhojpuri, Thami, Wambule, and Darai separately. Native-script targets such as Limbu, Sunuwar/Jenticha, Kirat Rai, Lepcha, Tirhuta, Gurung Khema, and Newa must stay separate from fallback or synthetic-only results. ## Current Implementation Slice - `tools/run_recognizer_eval.py` scores an existing recognizer prediction file against a PaddleOCR-style label file. It records non-empty model identity plus config/checkpoint/dictionary, SHA-256 identities for local checkpoint/config/dictionary artifacts and ties those identities back to the visible run fields, prediction, label, manifest, summary/datasheet, script inventory, train/dev manifest, command, machine platform/Python metadata, normalization, eval-pack version, frozen-pack declaration, selection declaration, per-row crop image SHA-256 identities and source-document/page/bbox provenance in `predictions.tsv` and `worst-lines.csv`, deterministic row- bootstrap 95% confidence intervals for codepoint CER, grapheme CER, and exact-line accuracy, observed and missing required script codepoint/grapheme inventory coverage, and claim-safety warnings, and emits a direct `claim-safety.md` report in the score directory. A claim-safe recognizer run now requires those artifacts to be readable local files whose hashes can be audited and at least 30 scored rows per emitted language/script/domain/GT/split slice so tiny or pooled smoke packs cannot pass as paper evidence. Declared manifest crop-image hashes must match actual crop bytes; missing script inventory files or missing required codepoints/graphemes are claim-safety blockers, and rows declared `human_verified` must carry human review evidence such as reviewer or review-artifact metadata. - `tools/audit_recognizer_eval_pack.py` audits a recognizer oracle-crop eval pack before scoring. It verifies label/image/manifest/summary integrity, duplicate labels, manifest identifiers, and manifest image paths/basenames, manifest text/hash consistency, source-document/ page/bbox coverage, language/script/domain/GT/split metadata, claim eligibility, frozen-pack declaration, unseen-for-selection declaration, and minimum scoring-unit counts overall and per language/script/domain/GT/split slice. - `tools/normalize_recognizer_eval_pack.py` converts existing PaddleOCR-style recognizer packs into a self-contained canonical pack with copied images, normalized `eval.txt`, `manifests/recognizer-eval.jsonl`, and `summary.json`. It preserves legacy provenance fields, fills only explicitly provided defaults, hashes copied images, and fails on missing images, duplicate labels, duplicate manifest IDs or image paths/basenames, or manifest/label text mismatches. - `configs/recognizer-eval-pack-normalization-v1.json` records reproducible normalization recipes for the current real recognizer packs: Newa, Sunuwar, Lepcha, Limbu, Kirat Rai, and shared Devanagari. `tools/run_recognizer_pack_normalization.py` validates those recipes, supports dry-runs, materializes selected packs through the normalizer, writes per-pack `audit/recognizer-eval-pack-audit.{json,md}` artifacts, and records pack audit results in `normalization-run.json`. - `configs/eval-target-coverage-v1.json` records the intended language/script coverage sets: the broader first-ten document OCR targets and the current real recognizer packs. `tools/validate_eval_target_coverage.py` compares those targets against the normalization and eval-run recipe registries so missing packs, missing eval recipes, non-claim-ready targets, and shared-pack language/script mismatches that could hide per-language failures are explicit gaps. Target `eval_status` values are constrained to known states, and only `registered_real_pack` can become coverage-ready; partial, shared, missing, or invented status labels cannot silently pass as claim support. - `tools/run_page_ocr_eval.py` scores saved page-level OCR `document.json` outputs against structured references. It emits an explicit `eval_mode=page_level_ocr` artifact, separates page OCR from recognizer oracle-crop metrics, records model/run provenance with local checkpoint/config/dictionary SHA-256 identities, records scored source image and `document.json` SHA-256 identities plus the embedded `document.source_path` in `page-metrics.csv`, rejects claim-safe runs whose `document.source_path` does not match the eval-manifest input, records and audits the page pack `pack_content_sha256`, recomputes page text micro CER over codepoints and grapheme clusters, audits `page-metrics.csv` row identity back to the eval manifest, resolved `document.json`, scorer metadata defaults, row claim eligibility, and parsed reference fields, reports auditable per-language/script/domain/GT/split page slices recomputed from `page-metrics.csv`, existing document metrics such as reading-order pair accuracy, table scores, figure scores, reconstruction metrics, page text exact accuracy, canonical page prediction/reference text pairs, per-row reference path/hash/trust artifacts, source-document/page/bbox provenance columns, reference coverage, missing document/reference rows, row-level source-document/page/bbox/ claim-eligibility and language/script/domain/GT metadata coverage, row-level claim eligibility, line-bbox IoU matching diagnostics when references provide explicit reading-order bboxes, worst-page review CSV/HTML artifacts, deterministic page-bootstrap 95% confidence intervals for page text CER, observed and missing required script codepoint/grapheme inventory coverage from reference text, and train/dev/eval leakage checks with recorded train-manifest hashes. Page GT declared as `machine_converted_spot_checked` requires an eligible native-reader calibration artifact, matching the recognizer-side trust gate. Page claim-safe runs require at least 30 scored pages per emitted language/script/domain/GT/split slice, and page score directories also emit `claim-safety.md` so they can be reviewed without a recipe wrapper. Duplicate page manifest sample IDs or image paths and stale manifest image hashes fail before scoring, and missing script inventory files or missing required codepoints/graphemes are claim-safety blockers. Page rows declared `human_verified` must also carry human review evidence rather than relying on a trust label alone. - `tools/run_eval_output_command.py` materializes canonical eval outputs through a recorded external command before scoring. In `recognizer` mode it writes one prediction file from a label pack; in `page` mode it runs once per eval-manifest row and requires each sample to produce `document.json`. It records the exact expanded command(s), command template, invocation, timestamp, stdout/stderr log paths, output hashes/sizes, per-page input image paths/hashes, model/checkpoint/config/dictionary metadata, machine details, and failure details with machine/invocation and label or eval-manifest hashes when a command does not produce the required artifact. When `--model-config` is provided without `--command`, it uses the repo's candidate engine defaults: recognizer mode calls `tools/predict_recognizer_eval.py`, while page mode calls `ocrtech parse-doc`. - `tools/predict_recognizer_eval.py` turns a candidate model card into a canonical recognizer prediction file for oracle-crop eval packs. It reads the label file, runs the audited `candidate` engine on each crop, writes `imageprediction confidence`, and records model-card, label, output, machine, and command hashes. - `tools/prepare_native_spotcheck_bundle.py` creates native-reader review bundles from recognizer labels, and `tools/apply_native_spotcheck_reviews.py` ingests completed review CSVs into calibration artifacts. The calibration step validates statuses, emits reviewed-label and manifest-override artifacts, measures machine-converted label CER against corrected/native-reviewed text, requires reviewer identity on reviewed rows, and records whether the spot-check is eligible evidence for treating a pack as `machine_converted_spot_checked`. Scorers bind that calibration evidence to the current run's language, script, domain, `machine_converted` source GT status, and `machine_converted_spot_checked` recommendation. Bundle summaries record source file hashes, machine metadata, timestamp, and review artifact paths. - `tools/build_recognizer_leakage_manifest.py` turns recognizer train/dev label files into JSONL leakage manifests with sample IDs, image hashes, text hashes, and combined sample hashes. These manifests are the standard input for scorer-side train/dev/eval leakage checks; text-only mode exists for corpora where image files are intentionally unavailable. Page-derived crop manifests must also preserve `source_sample_id`, and the page scorer/audit treat train `source_sample_id` overlap with eval page `sample_id` as hard leakage. - `docs/EVAL_HARNESS_RUNBOOK.md` gives the checked operational recipe for chaining normalization, leakage-manifest generation, output materialization, recognizer scoring, native-reader calibration, and page-level scoring. It documents the exact claim-safety flags and the distinction between repo candidate model-card runs and external PaddleOCR/HF command templates. - `configs/eval-harness-recipes-v1.json` records concrete current recipe instances for Newa, Sunuwar, Lepcha, Limbu, Kirat Rai, shared Devanagari, and packaged HF/Paddle recognizer smoke paths. `tools/validate_eval_harness_recipes.py` verifies that referenced local paths exist, runs the canonical recognizer eval-pack audit for each path-complete recognizer pack, statically audits page eval manifests for source-image hashes, page provenance, row claim eligibility, human-review evidence, and claim-ready references plus minimum overall and per-slice scoring-unit counts, and separates structural recipe validity from claim readiness blockers, including missing, malformed, or recipe-contradicting `summary_file` metadata for language/script/task/dataset/domain/GT/split/version/eligibility, missing or malformed `script_inventory_file`, incomplete row provenance, pack-audit warnings or issues, missing, malformed, ineligible, or mismatched recipe-level native spot-check calibration when default or audited row GT is `machine_converted_spot_checked`, scorer-supported normalization profile declarations, external command templates missing required input/output placeholders or containing unsupported placeholders, unresolved external prediction command templates, commands still marked as requiring operator resolution, unhashable checkpoint/config/dictionary artifacts, malformed or empty `leakage_sources`, declared checkpoint selection on the test pack, and declared known leakage. - `tools/run_eval_recipe.py` is the recipe-level recognizer orchestrator. It validates the selected recipe, builds or wires train/dev leakage manifests, materializes predictions through the recorded runner, scores the canonical predictions, and writes `recipe.json`, `recipe-validation.json`, `recipe-run.json`, root-level `audit.json`, root-level `claim-safety.md`, materialization logs, leakage artifacts, and the scorer output under one reviewable run directory. The root claim-safety report carries the score-mode-specific recognizer or page facts from the recipe, score, and audit JSON files, including script-inventory coverage, metadata contradictions, uncertainty intervals, leakage status, and the score-mode review artifact list, so it can be audited for stale reviewer-facing summaries, root `recipe.json` is tied back to the selected recipe in the hashed registry file, root `recipe-validation.json` is tied back to `recipe-run.json` and recomputed from the current validator, recipe-run timestamp/machine/command provenance is checked, root materialization/audit/claim-safety output pointers must point at the run directory's own artifacts, root audit summaries must preserve current score-audit issue lists, warnings, claim-safe status, and score-dir identity, root score summary paths and hashes must match `score/metrics.json` and `score/eval-run.json`, root train-manifest path/hash identities must match the score eval-run leakage inputs, the hashed recipe registry file must sit under the declared recipe root, and the root prediction/materialization paths, command template, output report, and materialized model/checkpoint/config/dictionary metadata are tied back to the scored prediction file hash and score eval-run metadata. Failed recipe attempts write `recipe-run-error.json` with timestamp, machine, command, recipe validation, registry hash, root, output directory, exception type, message, and traceback so a failed run is still attributable. - `tools/compare_recognizer_eval_runs.py` compares two completed recognizer oracle-crop score directories on the same frozen eval pack. It pairs canonical `predictions.tsv` rows by `sample_id`, verifies eval-pack/run identity including `pack_content_sha256` and row GT identity, emits `comparison.json`, `comparison.md`, `paired-rows.tsv`, and `command.txt`, reports candidate and baseline micro CER over codepoints and graphemes, paired wins/losses/ties over row-level CER, deterministic paired-row bootstrap confidence intervals for the CER improvement, and an exact two-sided sign-test p-value. Its audit mode checks run provenance, recomputes the comparison, and rejects stale JSON, Markdown, command, or paired-row artifacts. - `tools/compare_page_ocr_eval_runs.py` does the same for completed page-level OCR score directories, using `page-eval-run.json`, `page-metrics.csv`, and `text-pairs.tsv`. It verifies page eval-pack identity and page/reference text row identity including `pack_content_sha256`, emits `comparison.json`, `comparison.md`, `paired-pages.tsv`, and `command.txt`, reports paired page text codepoint/grapheme CER improvements, deterministic paired-page bootstrap intervals, and an exact two-sided sign-test p-value, and can audit command/provenance plus the comparison artifacts back to the current score directories. - `tools/audit_eval_run.py` audits completed recognizer or page score directories, or a recipe-run root with `score/`, for required artifacts, run metadata, metric arithmetic, timestamp and command-file consistency, machine platform/Python provenance, recorded input-file hash consistency, scorer-supported normalization profile consistency, normalization-change summary recomputation from recognizer `predictions.tsv` or page `text-pairs.tsv`, model-artifact hash and run-field consistency, recognizer `predictions.tsv` source-row recomputation from the recorded label/prediction/manifest files, row-level edit-distance and per-language/script/domain/GT/split slice recomputation, recognizer crop-image and manifest-declared image-hash verification, recognizer source label/prediction recomputation for label-line, matched-line, unmatched-label, unmatched-prediction, match-rate metrics, deterministic bootstrap uncertainty intervals, and script inventory recomputation from recognizer GT rows, recognizer `unmatched.tsv` identity/hash/score recomputation from the source label and prediction files, recognizer `confusions.json` recomputation from canonical predictions, page `reference-artifacts.tsv` recomputation from scored reference files, page `text-pairs.tsv` recomputation from scored document/reference text, page `confusions.json` recomputation from those canonical text pairs, page text CER row and total recomputation from canonical page text pairs, page row provenance recomputation from source manifests, scored source-image/ `document.json` hash verification, page bootstrap uncertainty interval recomputation, script inventory recomputation from page reference text, and page-metric identity recomputation from manifest/document/reference sources, run-level language/script/domain/GT/split list recomputation from canonical row artifacts, page `line-alignment.csv` row-content and detection/alignment recomputation from scored document/reference files, including predicted/reference line IDs, bboxes, text, IoU, and per-line text CER, page recognizer normalizer/defaulted-metadata count checks against the recorded summary/datasheet, direct-run summary/datasheet metadata consistency checks against run language, script, domain, GT, split, eval-pack version, frozen/unseen, default claim-eligibility, and task fields, dictionary coverage recomputation from recognizer GT rows or page reference text, with GT/reference codepoints outside the recorded dictionary treated as a claim-safety blocker, `missing.tsv` recomputation from the eval manifest, document outputs, and reference files, slice totals, duplicate scored sample/image IDs in completed output artifacts, ambiguous recognizer manifest image keys, duplicate page manifest image paths, page manifest image-hash mismatches, leakage-check presence and leakage-overlap recomputation from train/eval sources, native spot-check calibration summary checks against the recorded calibration artifact, metadata-contradiction recomputation from source provenance signals, provenance/metadata coverage recomputation from source manifests, page reference-coverage and manifest/scored/missing/claim-eligible count recomputation from source artifacts, durable claim-safety blockers such as metadata contradictions, missing manifests or summary/datasheets, unrun leakage checks, leakage overlaps, low match rate, unsafe domains, untrusted GT, missing provenance, missing uncertainty evidence, too-small claim-facing sample size, missing script inventory evidence, missing required script codepoints/graphemes, zero-length recognizer GT rows or page text references, recognizer row-level claim ineligibility, and incomplete page coverage, warning/claim-safety consistency, `metrics.md` consistency against `metrics.json`, recomputed worst-example CSVs, reviewer CSV order/fields, and reviewer HTML contents, direct score-directory claim reports whose core lines are checked against JSON run facts, recipe-root claim reports whose recognizer/page score facts are checked against the nested score directory, and materialization artifacts whose timestamp, machine platform/Python and invocation metadata, JSON records, model/checkpoint/config/dictionary metadata, static recipe-level dictionary coverage checks against recognizer GT or page reference codepoints, per-record commands/log paths, command-template file, Markdown report, recognizer prediction path/hash or page document-output directory, and output hashes are checked against the score eval-run facts plus claim-report artifacts whose top-level report is also checked against recipe and score JSON facts. - The scorer computes corpus micro CER over NFC codepoints and Unicode extended grapheme clusters, exact-line accuracy, matched/unmatched rows, per-language/script/ domain/GT/split slice metrics, worst lines/pages, review CSV/HTML, and recognizer/page confusion counts. - Claim-safety warnings now cover missing manifest or summary, missing eval-pack version, missing script inventory files, missing required script codepoints or graphemes, GT/reference codepoints outside the recorded dictionary, unhashable model artifacts, zero-length recognizer GT rows or page text references, non-frozen or selection-used eval packs, low prediction match rate, mixed languages/scripts/domains/GT statuses/splits, unsafe domains, untrusted GT, metadata contradictions such as synthetic/transliterated provenance declared as real material or machine-converted provenance declared as `human_verified`, missing source-document/page/bbox/claim-eligibility provenance, explicit row-level claim ineligibility, missing page row source-document/page/bbox/ claim-eligibility/language/script/domain/GT metadata, normalizer-filled metadata defaults from canonical pack summaries, missing human-review evidence for `human_verified` rows, missing, mismatched, or ineligible native spot-check calibration when `machine_converted_spot_checked` is claimed, known leakage, unrun leakage checks, and train/dev/eval overlaps from provided JSONL manifests. - A Newa recognizer artifact was produced at `outputs/evals/newa-prachalit/newa-native-rescore-2026-06-19-zenodo/`. It is intentionally **not** claim-safe: GT is machine-converted, the pack is not declared unseen for selection, page provenance and claim eligibility are missing, and no train/dev leakage manifest was provided. - The same Newa eval pack has a standalone audit at `outputs/evals/newa-prachalit/newa-real-eval-pack-audit-2026-06-19/`. Basic pack integrity passes (`3180/3180` labels matched to manifest rows and images), but claim readiness fails because the pack is not declared unseen for selection, has no page metadata, lacks row-level claim eligibility, and uses machine-converted GT. ## Remaining Gaps - Resolve remaining remote-only checkpoint/package locations in `configs/eval-harness-recipes-v1.json` and replace placeholder external commands with runnable per-family commands. The recipe registry exists and validates local paths, and `tools/run_eval_recipe.py` can execute resolved recognizer recipes, but several language-specific PaddleOCR checkpoints are not packaged locally or hash-verifiable, so the validator must keep them out of claim-ready status. - Extend page-level evaluation with live detector output capture and crop lineage from model runners; the current page harness scores saved structured documents and emits per-line IoU/alignment rows when references provide explicit line bboxes, but it does not yet run the detector/recognizer pipeline itself. - Archive and review the materialized canonical eval packs under `outputs/eval-packs/`. The six current real recognizer packs now exist with per-pack audits, but all six are structurally non-claim-ready because of machine-converted GT, missing or defaulted provenance/claim metadata, and for some packs selection-status blockers. - Fill the target coverage gaps reported by `tools/validate_eval_target_coverage.py`. The current coverage config validates, but is intentionally not coverage-ready: English, Tamang, Magar, Wambule, Bantawa/Rai, Kulung, Yakkha, and Tharu/Rana Tharu still lack registered real eval packs, and the current registered real-pack recipes remain non-claim-ready because model commands, GT trust, provenance, calibration, or row eligibility blockers remain unresolved. A target also remains non-coverage-ready unless its `eval_status` is exactly `registered_real_pack`; use `partial_real_recognizer_pack`, `shared_devanagari_pack`, or `missing_real_eval_pack` only when the resulting gap is intended and visible. - Generate and wire train/dev leakage manifests for each actual recognizer family and pass them to scoring runs. The manifest builder exists, but each family still needs its concrete train/dev label paths recorded in the run recipes, including page-origin `source_sample_id` for crops selected from full-page eval material. - Apply native-reader spot-check/calibration to the actual language eval packs, with enough reviewed rows per script/domain to justify any `machine_converted_spot_checked` upgrade. The harness now has artifacts and scorer wiring for this, but the reviewed human evidence still has to be collected. ## Research Basis - OCR-D states that OCR quality evaluation requires representative ground truth, realistic images, manual transcriptions, localization of errors by document/page/ region, and repeatable workflows: - The OCR evaluation survey by Neudecker et al. describes CER as edit-distance errors divided by reference character count and highlights that OCR evaluation depends on alignment, reading order, layout, and tool-specific normalization choices: - Unicode UAX #29 defines extended grapheme clusters as the default programmatic approximation of user-perceived characters, which is necessary for scripts where one visible unit can span multiple codepoints: - PaddleOCR's `RecMetric` implementation averages normalized Levenshtein distance per sample and returns `norm_edit_dis`; this is not the same quantity as micro CER: - Datasheets for Datasets motivates documenting dataset motivation, composition, collection, and recommended use, which maps directly onto eval-pack provenance: - Model Cards for Model Reporting motivates model documentation with benchmarked evaluation, intended use, limitations, and evaluation procedure: - Recent OCR evaluation survey work warns that benchmark design can make historical and marginalized documents invisible when evaluation emphasizes convenient modern layouts rather than the actual target material: