|
Download ocr-tech/docs/EVAL_HARNESS_GOAL.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 32.1 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/EVAL_HARNESS_GOAL.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/EVAL_HARNESS_GOAL.md
-
curl -L -o EVAL_HARNESS_GOAL.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/EVAL_HARNESS_GOAL.md
32.1 kB
| # Evaluation Harness Goal | |
| Updated: 2026-06-20 | |
| ## Current Status | |
| This is a parked claim-safety design, not the active product work. The active | |
| 2026-06-23 objective is Limbu book OCR: real book pages in, editable Unicode | |
| book text out, with line structure preserved and unresolved text surfaced for | |
| human review. Do not add generic eval-harness features from this document unless | |
| they directly unblock that OCR loop. For the current Limbu work, the useful | |
| evaluation surface is deliberately thin: frozen real-book page acceptance checks, | |
| artifact audits, page/line counts, missed/extra/blank-line rates, review-packet | |
| coverage, and reviewed-only training-pack export validation. | |
| ## Goal | |
| The goal is a claim-grade OCR evaluation harness for the Nepal/diaspora recognizer | |
| suite that makes weak publication claims difficult by default, not merely inconvenient: | |
| each language/script result must be scored only from a frozen, versioned eval pack whose | |
| source images, labels, manifest rows, source-document/page/bbox provenance, split role, | |
| domain, ground-truth trust evidence, claim eligibility, canonical pack-content digest, | |
| and script/grapheme coverage are | |
| auditable; each run must record the exact model, checkpoint, config, dictionary, | |
| prediction file, normalization profile, command, timestamp, machine, pack-content digest, | |
| and artifact hashes; | |
| each metric must be recomputed from canonical predictions as corpus micro edit-distance | |
| CER over both Unicode codepoints and UAX #29 extended grapheme clusters, with exact-line | |
| accuracy, matched/unmatched rows, worst examples, confidence intervals, and paired | |
| comparison tests when claims compare systems; recognizer-only oracle-crop scores must | |
| remain separate from page-level OCR scores that include detection, crop lineage, reading | |
| order, and alignment; and the harness must fail or hard-warn on missing images, duplicate | |
| IDs, low match rate, train/dev/test leakage, checkpoint selection on test data, synthetic | |
| or transliterated data presented as real evidence, machine-converted labels presented as | |
| human-verified evidence, human GT without review/agreement evidence, eval packs too small | |
| for the claimed conclusion, uncovered script inventories, and aggregate averages that | |
| hide per-language, per-script, per-domain, or per-GT-status failures. A reviewer opening | |
| one output directory should be able to decide what was scored, against which ground truth, | |
| how trustworthy that ground truth is, whether the test set was frozen and unseen for | |
| selection, what normalization and metric were used, how uncertain the number is, what the | |
| failures look like, what script material was not covered, and whether the result is safe | |
| for a paper claim. | |
| ## Requirements | |
| - **Eval packs are versioned evidence objects.** Each pack must include `images/`, | |
| a label file, a manifest JSONL, and a summary/datasheet containing source documents, | |
| page/bbox provenance, language, script, domain, split role, GT source, GT trust | |
| level, known leakage status, claim eligibility, and a `pack_content_sha256` over the | |
| canonical row identity/text/provenance fields so pack drift is detectable across | |
| normalization, scoring, audit, and comparison. | |
| - **Metrics are canonical and micro-averaged.** The harness must compute | |
| total-edit-distance / total-reference-length CER, not per-line macro averages. | |
| It must report both codepoint CER and grapheme-cluster CER, plus exact-line accuracy, | |
| matched/unmatched counts, prediction match rate, total reference units, and | |
| worst-line examples. | |
| - **Point estimates are not enough for claims.** Every claim-facing metric must include | |
| confidence intervals computed by resampling the scoring units, and same-pack system | |
| comparisons must include a paired test over per-row errors. The report must show the | |
| effective sample size per language/script/domain and hard-warn when a pack is too | |
| small for the claimed conclusion. | |
| - **PaddleOCR training metrics are not claim metrics.** PaddleOCR `RecMetric` reports | |
| exact-line `acc` and `norm_edit_dis` as one minus the average per-sample normalized | |
| Levenshtein distance; this may be useful during training, but it is not the | |
| corpus-level micro CER used for claims. | |
| - **Recognition and page OCR are separate tasks.** Oracle-crop recognizer results are | |
| valid recognizer evidence only. Page-level OCR claims require page images, detector | |
| output, crop/reading-order/alignment logic, and a separate end-to-end score. | |
| - **Unicode handling is explicit.** Text is normalized with a named profile, defaulting | |
| to NFC with whitespace preserved. The audit must reject a stale or unsupported | |
| profile that no longer matches the metric recomputation path. Metrics must also | |
| report how many raw GT/reference and prediction strings changed under that profile, | |
| and the audit must recompute those counts from canonical row artifacts so hidden | |
| normalization effects cannot disappear into a single CER. Grapheme metrics use | |
| Unicode extended grapheme clusters; any script-specific tailoring must be recorded. | |
| - **Human verification is first-class metadata.** Machine-converted GT may be scored, | |
| but the output must label it as machine-converted. Paper-grade human-reading claims | |
| require either fully reviewed GT or a documented native-reader calibration sample. | |
| Human-verified GT must record who reviewed it, how much was double-reviewed, and the | |
| observed disagreement or correction rate; a trust label without review evidence is not | |
| claim evidence. | |
| - **Script coverage is evidence, not decoration.** Each eval pack must report the | |
| codepoint, grapheme-cluster, digit, punctuation, conjunct/combining-mark, and | |
| language-specific inventory coverage it exercises, plus gaps against the intended | |
| script inventory. A low CER on a pack that omits common graphemes or conjuncts must not | |
| be eligible for broad script support claims. | |
| - **Reports are inspectable.** Each run should emit `eval-run.json`, `metrics.json`, | |
| `metrics.md`, `predictions.tsv`, source-audited `unmatched.tsv`, `worst-lines.csv`, | |
| `worst-pages.csv`, `sample-review.csv`, `sample-review.html`, `confusions.json`, `command.txt`, and a | |
| human-readable `claim-safety.md` report. | |
| Worst-example rows must carry the row identity, hashes, language/script/domain/GT/split, | |
| claim eligibility, source-document/page/bbox provenance, and reference-artifact trust | |
| fields needed to inspect a failure without reverse-engineering the rest of the run. | |
| - **Reporting is per language/script/domain.** Shared recognizers, such as the | |
| Devanagari model, must still report Majhi, Magar, Sunuwar-Devanagari, Magahi, | |
| Bhojpuri, Thami, Wambule, and Darai separately. Native-script targets such as | |
| Limbu, Sunuwar/Jenticha, Kirat Rai, Lepcha, Tirhuta, Gurung Khema, and Newa must | |
| stay separate from fallback or synthetic-only results. | |
| ## Current Implementation Slice | |
| - `tools/run_recognizer_eval.py` scores an existing recognizer prediction file | |
| against a PaddleOCR-style label file. It records non-empty model identity plus config/checkpoint/dictionary, | |
| SHA-256 identities for local checkpoint/config/dictionary artifacts and ties those | |
| identities back to the visible run fields, prediction, label, manifest, | |
| summary/datasheet, script inventory, train/dev manifest, command, machine platform/Python metadata, | |
| normalization, eval-pack version, frozen-pack declaration, selection declaration, | |
| per-row crop image SHA-256 identities and source-document/page/bbox provenance in | |
| `predictions.tsv` and `worst-lines.csv`, deterministic row- | |
| bootstrap 95% confidence intervals for codepoint CER, grapheme CER, and exact-line | |
| accuracy, observed and missing required script codepoint/grapheme inventory coverage, | |
| and claim-safety warnings, and emits a direct `claim-safety.md` report in | |
| the score directory. A claim-safe recognizer run now requires those artifacts to be | |
| readable local files whose hashes can be audited and at least 30 scored rows per | |
| emitted language/script/domain/GT/split slice so tiny or pooled smoke packs cannot | |
| pass as paper evidence. Declared manifest crop-image hashes must | |
| match actual crop bytes; missing script inventory files or missing required | |
| codepoints/graphemes are claim-safety blockers, and rows declared `human_verified` | |
| must carry human review evidence such as reviewer or review-artifact metadata. | |
| - `tools/audit_recognizer_eval_pack.py` audits a recognizer oracle-crop eval pack | |
| before scoring. It verifies label/image/manifest/summary integrity, duplicate | |
| labels, manifest identifiers, and manifest image paths/basenames, manifest text/hash consistency, source-document/ | |
| page/bbox coverage, language/script/domain/GT/split metadata, claim eligibility, | |
| frozen-pack declaration, unseen-for-selection declaration, and minimum scoring-unit | |
| counts overall and per language/script/domain/GT/split slice. | |
| - `tools/normalize_recognizer_eval_pack.py` converts existing PaddleOCR-style | |
| recognizer packs into a self-contained canonical pack with copied images, normalized | |
| `eval.txt`, `manifests/recognizer-eval.jsonl`, and `summary.json`. It preserves legacy | |
| provenance fields, fills only explicitly provided defaults, hashes copied images, and | |
| fails on missing images, duplicate labels, duplicate manifest IDs or image | |
| paths/basenames, or manifest/label text mismatches. | |
| - `configs/recognizer-eval-pack-normalization-v1.json` records reproducible | |
| normalization recipes for the current real recognizer packs: Newa, Sunuwar, Lepcha, | |
| Limbu, Kirat Rai, and shared Devanagari. `tools/run_recognizer_pack_normalization.py` | |
| validates those recipes, supports dry-runs, materializes selected packs through the | |
| normalizer, writes per-pack `audit/recognizer-eval-pack-audit.{json,md}` artifacts, | |
| and records pack audit results in `normalization-run.json`. | |
| - `configs/eval-target-coverage-v1.json` records the intended language/script coverage | |
| sets: the broader first-ten document OCR targets and the current real recognizer | |
| packs. `tools/validate_eval_target_coverage.py` compares those targets against the | |
| normalization and eval-run recipe registries so missing packs, missing eval recipes, | |
| non-claim-ready targets, and shared-pack language/script mismatches that could hide | |
| per-language failures are explicit gaps. Target `eval_status` values are constrained | |
| to known states, and only `registered_real_pack` can become coverage-ready; partial, | |
| shared, missing, or invented status labels cannot silently pass as claim support. | |
| - `tools/run_page_ocr_eval.py` scores saved page-level OCR `document.json` outputs | |
| against structured references. It emits an explicit `eval_mode=page_level_ocr` | |
| artifact, separates page OCR from recognizer oracle-crop metrics, records model/run | |
| provenance with local checkpoint/config/dictionary SHA-256 identities, records | |
| scored source image and `document.json` SHA-256 identities plus the embedded | |
| `document.source_path` in `page-metrics.csv`, rejects claim-safe runs whose | |
| `document.source_path` does not match the eval-manifest input, records and audits | |
| the page pack `pack_content_sha256`, recomputes page text micro CER over codepoints | |
| and grapheme clusters, | |
| audits `page-metrics.csv` row identity back to the eval manifest, resolved | |
| `document.json`, scorer metadata defaults, row claim eligibility, and parsed | |
| reference fields, reports | |
| auditable per-language/script/domain/GT/split page slices recomputed from | |
| `page-metrics.csv`, existing document metrics such as reading-order pair accuracy, | |
| table scores, figure scores, reconstruction | |
| metrics, page text exact accuracy, canonical page prediction/reference text | |
| pairs, per-row reference path/hash/trust artifacts, source-document/page/bbox provenance columns, reference coverage, | |
| missing document/reference rows, row-level source-document/page/bbox/ | |
| claim-eligibility and language/script/domain/GT metadata coverage, row-level | |
| claim eligibility, line-bbox IoU matching diagnostics when | |
| references provide explicit reading-order bboxes, | |
| worst-page review CSV/HTML artifacts, deterministic page-bootstrap 95% confidence | |
| intervals for page text CER, observed and missing required script codepoint/grapheme | |
| inventory coverage from reference text, and train/dev/eval leakage checks with recorded | |
| train-manifest hashes. Page GT declared as `machine_converted_spot_checked` requires | |
| an eligible native-reader calibration artifact, matching the recognizer-side trust | |
| gate. Page claim-safe runs require at least 30 scored pages per emitted | |
| language/script/domain/GT/split slice, and page score | |
| directories also emit `claim-safety.md` so they can be reviewed without a recipe | |
| wrapper. Duplicate page manifest sample IDs or image paths and stale manifest image | |
| hashes fail before scoring, and missing script inventory files or missing required | |
| codepoints/graphemes are claim-safety blockers. Page rows declared `human_verified` | |
| must also carry human review evidence rather than relying on a trust label alone. | |
| - `tools/run_eval_output_command.py` materializes canonical eval outputs through a | |
| recorded external command before scoring. In `recognizer` mode it writes one | |
| prediction file from a label pack; in `page` mode it runs once per eval-manifest row | |
| and requires each sample to produce `document.json`. It records the exact expanded | |
| command(s), command template, invocation, timestamp, stdout/stderr log paths, output | |
| hashes/sizes, per-page input image paths/hashes, model/checkpoint/config/dictionary metadata, machine details, and | |
| failure details with machine/invocation and label or eval-manifest hashes when a | |
| command does not produce the required artifact. When | |
| `--model-config` is provided without | |
| `--command`, it uses the repo's candidate engine defaults: recognizer mode calls | |
| `tools/predict_recognizer_eval.py`, while page mode calls `ocrtech parse-doc`. | |
| - `tools/predict_recognizer_eval.py` turns a candidate model card into a canonical | |
| recognizer prediction file for oracle-crop eval packs. It reads the label file, | |
| runs the audited `candidate` engine on each crop, writes `image<TAB>prediction<TAB> | |
| confidence`, and records model-card, label, output, machine, and command hashes. | |
| - `tools/prepare_native_spotcheck_bundle.py` creates native-reader review bundles from | |
| recognizer labels, and `tools/apply_native_spotcheck_reviews.py` ingests completed | |
| review CSVs into calibration artifacts. The calibration step validates statuses, | |
| emits reviewed-label and manifest-override artifacts, measures machine-converted label | |
| CER against corrected/native-reviewed text, requires reviewer identity on reviewed | |
| rows, and records whether the spot-check is eligible evidence for treating a pack as | |
| `machine_converted_spot_checked`. Scorers bind that calibration evidence to the | |
| current run's language, script, domain, `machine_converted` source GT status, and | |
| `machine_converted_spot_checked` recommendation. Bundle summaries record source file | |
| hashes, machine metadata, timestamp, and review artifact paths. | |
| - `tools/build_recognizer_leakage_manifest.py` turns recognizer train/dev label files | |
| into JSONL leakage manifests with sample IDs, image hashes, text hashes, and combined | |
| sample hashes. These manifests are the standard input for scorer-side train/dev/eval | |
| leakage checks; text-only mode exists for corpora where image files are intentionally | |
| unavailable. Page-derived crop manifests must also preserve `source_sample_id`, and | |
| the page scorer/audit treat train `source_sample_id` overlap with eval page | |
| `sample_id` as hard leakage. | |
| - `docs/EVAL_HARNESS_RUNBOOK.md` gives the checked operational recipe for chaining | |
| normalization, leakage-manifest generation, output materialization, recognizer scoring, | |
| native-reader calibration, and page-level scoring. It documents the exact claim-safety | |
| flags and the distinction between repo candidate model-card runs and external | |
| PaddleOCR/HF command templates. | |
| - `configs/eval-harness-recipes-v1.json` records concrete current recipe instances for | |
| Newa, Sunuwar, Lepcha, Limbu, Kirat Rai, shared Devanagari, and packaged HF/Paddle | |
| recognizer smoke paths. `tools/validate_eval_harness_recipes.py` verifies that | |
| referenced local paths exist, runs the canonical recognizer eval-pack audit for each | |
| path-complete recognizer pack, statically audits page eval manifests for source-image | |
| hashes, page provenance, row claim eligibility, human-review evidence, and | |
| claim-ready references plus minimum overall and per-slice scoring-unit counts, and | |
| separates structural recipe validity from claim readiness | |
| blockers, including missing, malformed, or recipe-contradicting `summary_file` metadata | |
| for language/script/task/dataset/domain/GT/split/version/eligibility, | |
| missing or malformed `script_inventory_file`, incomplete row provenance, | |
| pack-audit warnings or issues, missing, malformed, ineligible, or mismatched recipe-level | |
| native spot-check calibration when default or audited row GT is `machine_converted_spot_checked`, | |
| scorer-supported normalization profile declarations, | |
| external command templates missing required input/output placeholders or containing | |
| unsupported placeholders, unresolved external prediction command templates, commands still marked as | |
| requiring operator resolution, unhashable checkpoint/config/dictionary artifacts, | |
| malformed or empty `leakage_sources`, declared | |
| checkpoint selection on the test pack, and declared known leakage. | |
| - `tools/run_eval_recipe.py` is the recipe-level recognizer orchestrator. It validates | |
| the selected recipe, builds or wires train/dev leakage manifests, materializes | |
| predictions through the recorded runner, scores the canonical predictions, and writes | |
| `recipe.json`, `recipe-validation.json`, `recipe-run.json`, root-level `audit.json`, | |
| root-level `claim-safety.md`, materialization logs, leakage artifacts, and the scorer | |
| output under one reviewable run directory. The root claim-safety report carries the | |
| score-mode-specific recognizer or page facts from the recipe, score, and audit JSON | |
| files, including script-inventory coverage, metadata contradictions, uncertainty | |
| intervals, leakage status, and the score-mode review artifact list, so it can be | |
| audited for stale reviewer-facing summaries, root `recipe.json` is tied back to the selected | |
| recipe in the hashed registry file, root `recipe-validation.json` is tied back to | |
| `recipe-run.json` and recomputed from the current validator, recipe-run | |
| timestamp/machine/command provenance is checked, root | |
| materialization/audit/claim-safety output pointers must point at the run directory's | |
| own artifacts, root audit summaries must preserve current score-audit issue lists, | |
| warnings, claim-safe status, and score-dir identity, root score summary paths and | |
| hashes must match `score/metrics.json` and `score/eval-run.json`, root train-manifest | |
| path/hash identities must match the score eval-run leakage inputs, the hashed recipe | |
| registry file must sit under the declared recipe root, and the root | |
| prediction/materialization paths, command template, output report, and materialized | |
| model/checkpoint/config/dictionary metadata are tied back to the scored prediction | |
| file hash and score eval-run metadata. Failed recipe attempts write | |
| `recipe-run-error.json` with timestamp, machine, command, recipe validation, | |
| registry hash, root, output directory, exception type, message, and traceback so a | |
| failed run is still attributable. | |
| - `tools/compare_recognizer_eval_runs.py` compares two completed recognizer oracle-crop | |
| score directories on the same frozen eval pack. It pairs canonical `predictions.tsv` | |
| rows by `sample_id`, verifies eval-pack/run identity including `pack_content_sha256` | |
| and row GT identity, emits | |
| `comparison.json`, `comparison.md`, `paired-rows.tsv`, and `command.txt`, reports candidate and | |
| baseline micro CER over codepoints and graphemes, paired wins/losses/ties over | |
| row-level CER, deterministic paired-row bootstrap confidence intervals for the CER | |
| improvement, and an exact two-sided sign-test p-value. Its audit mode checks run | |
| provenance, recomputes the comparison, and rejects stale JSON, Markdown, command, | |
| or paired-row artifacts. | |
| - `tools/compare_page_ocr_eval_runs.py` does the same for completed page-level OCR | |
| score directories, using `page-eval-run.json`, `page-metrics.csv`, and | |
| `text-pairs.tsv`. It verifies page eval-pack identity and page/reference text row | |
| identity including `pack_content_sha256`, emits `comparison.json`, | |
| `comparison.md`, `paired-pages.tsv`, and `command.txt`, reports | |
| paired page text codepoint/grapheme CER improvements, deterministic paired-page | |
| bootstrap intervals, and an exact two-sided sign-test p-value, and can audit | |
| command/provenance plus the comparison artifacts back to the current score | |
| directories. | |
| - `tools/audit_eval_run.py` audits completed recognizer or page score directories, or a | |
| recipe-run root with `score/`, for required artifacts, run metadata, metric arithmetic, | |
| timestamp and command-file consistency, machine platform/Python provenance, recorded | |
| input-file hash consistency, scorer-supported normalization profile consistency, | |
| normalization-change summary recomputation from recognizer `predictions.tsv` or page | |
| `text-pairs.tsv`, | |
| model-artifact hash and run-field consistency, | |
| recognizer `predictions.tsv` source-row recomputation from the recorded | |
| label/prediction/manifest files, row-level | |
| edit-distance and per-language/script/domain/GT/split slice recomputation, | |
| recognizer crop-image and manifest-declared image-hash verification, recognizer source label/prediction | |
| recomputation for label-line, matched-line, unmatched-label, unmatched-prediction, | |
| match-rate metrics, deterministic bootstrap uncertainty intervals, and script | |
| inventory recomputation from recognizer GT rows, recognizer | |
| `unmatched.tsv` identity/hash/score recomputation from the source | |
| label and prediction files, recognizer `confusions.json` recomputation from canonical | |
| predictions, page `reference-artifacts.tsv` recomputation from scored reference | |
| files, page `text-pairs.tsv` recomputation from scored document/reference text, | |
| page `confusions.json` recomputation from those canonical text pairs, | |
| page text CER row and total | |
| recomputation from canonical page text pairs, page row provenance recomputation | |
| from source manifests, scored source-image/ | |
| `document.json` hash verification, page bootstrap uncertainty interval recomputation, | |
| script inventory recomputation from page reference text, | |
| and page-metric identity recomputation from manifest/document/reference sources, | |
| run-level language/script/domain/GT/split | |
| list recomputation from canonical row artifacts, page | |
| `line-alignment.csv` row-content and | |
| detection/alignment recomputation from scored document/reference files, including | |
| predicted/reference line IDs, bboxes, text, IoU, and per-line text CER, page | |
| recognizer normalizer/defaulted-metadata count checks against the recorded | |
| summary/datasheet, | |
| direct-run summary/datasheet metadata consistency checks against run language, | |
| script, domain, GT, split, eval-pack version, frozen/unseen, default | |
| claim-eligibility, and task fields, | |
| dictionary coverage recomputation from recognizer GT rows or page reference text, | |
| with GT/reference codepoints outside the recorded dictionary treated as a | |
| claim-safety blocker, | |
| `missing.tsv` recomputation from the eval | |
| manifest, document outputs, and reference files, slice totals, duplicate scored sample/image IDs | |
| in completed output artifacts, ambiguous recognizer manifest image keys, | |
| duplicate page manifest image paths, page manifest | |
| image-hash mismatches, leakage-check presence and leakage-overlap recomputation | |
| from train/eval sources, native spot-check calibration summary checks against the | |
| recorded calibration artifact, metadata-contradiction recomputation from source | |
| provenance signals, provenance/metadata coverage recomputation from source | |
| manifests, page reference-coverage and manifest/scored/missing/claim-eligible count | |
| recomputation from source artifacts, durable | |
| claim-safety blockers | |
| such as metadata contradictions, missing manifests or summary/datasheets, unrun | |
| leakage checks, leakage overlaps, low match rate, unsafe domains, untrusted GT, | |
| missing provenance, missing uncertainty evidence, too-small claim-facing sample size, | |
| missing script inventory evidence, missing required script codepoints/graphemes, | |
| zero-length recognizer GT rows or page text references, recognizer row-level claim | |
| ineligibility, and incomplete page coverage, | |
| warning/claim-safety | |
| consistency, `metrics.md` consistency against `metrics.json`, recomputed | |
| worst-example CSVs, reviewer CSV order/fields, and reviewer HTML contents, direct | |
| score-directory claim reports whose core lines are checked against JSON run facts, | |
| recipe-root claim reports whose recognizer/page score facts are checked against the | |
| nested score directory, and materialization artifacts whose timestamp, machine platform/Python | |
| and invocation metadata, JSON records, model/checkpoint/config/dictionary metadata, | |
| static recipe-level dictionary coverage checks against recognizer GT or page | |
| reference codepoints, | |
| per-record commands/log paths, command-template file, | |
| Markdown report, recognizer prediction path/hash or page document-output directory, | |
| and output hashes are checked against the score | |
| eval-run facts plus claim-report artifacts whose top-level report is also checked | |
| against recipe and score JSON facts. | |
| - The scorer computes corpus micro CER over NFC codepoints and Unicode extended | |
| grapheme clusters, exact-line accuracy, matched/unmatched rows, per-language/script/ | |
| domain/GT/split slice metrics, worst lines/pages, review CSV/HTML, and recognizer/page | |
| confusion counts. | |
| - Claim-safety warnings now cover missing manifest or summary, missing eval-pack | |
| version, missing script inventory files, missing required script codepoints or | |
| graphemes, GT/reference codepoints outside the recorded dictionary, unhashable model | |
| artifacts, zero-length recognizer GT rows or page text references, non-frozen or selection-used eval packs, low | |
| prediction match rate, mixed languages/scripts/domains/GT statuses/splits, unsafe | |
| domains, untrusted GT, metadata contradictions such as synthetic/transliterated | |
| provenance declared as real material or machine-converted provenance declared as | |
| `human_verified`, missing source-document/page/bbox/claim-eligibility provenance, | |
| explicit row-level claim ineligibility, missing page row source-document/page/bbox/ | |
| claim-eligibility/language/script/domain/GT metadata, normalizer-filled metadata defaults from | |
| canonical pack summaries, missing human-review evidence for `human_verified` rows, missing, | |
| mismatched, or ineligible native spot-check calibration when | |
| `machine_converted_spot_checked` is claimed, known leakage, unrun leakage checks, and | |
| train/dev/eval overlaps from provided JSONL manifests. | |
| - A Newa recognizer artifact was produced at | |
| `outputs/evals/newa-prachalit/newa-native-rescore-2026-06-19-zenodo/`. It is | |
| intentionally **not** claim-safe: GT is machine-converted, the pack is not declared | |
| unseen for selection, page provenance and claim eligibility are missing, and no | |
| train/dev leakage manifest was provided. | |
| - The same Newa eval pack has a standalone audit at | |
| `outputs/evals/newa-prachalit/newa-real-eval-pack-audit-2026-06-19/`. Basic pack | |
| integrity passes (`3180/3180` labels matched to manifest rows and images), but | |
| claim readiness fails because the pack is not declared unseen for selection, has | |
| no page metadata, lacks row-level claim eligibility, and uses machine-converted GT. | |
| ## Remaining Gaps | |
| - Resolve remaining remote-only checkpoint/package locations in | |
| `configs/eval-harness-recipes-v1.json` and replace placeholder external commands with | |
| runnable per-family commands. The recipe registry exists and validates local paths, | |
| and `tools/run_eval_recipe.py` can execute resolved recognizer recipes, but several | |
| language-specific PaddleOCR checkpoints are not packaged locally or hash-verifiable, | |
| so the validator must keep them out of claim-ready status. | |
| - Extend page-level evaluation with live detector output capture and crop lineage from | |
| model runners; the current page harness scores saved structured documents and emits | |
| per-line IoU/alignment rows when references provide explicit line bboxes, but it does | |
| not yet run the detector/recognizer pipeline itself. | |
| - Archive and review the materialized canonical eval packs under `outputs/eval-packs/`. | |
| The six current real recognizer packs now exist with per-pack audits, but all six are | |
| structurally non-claim-ready because of machine-converted GT, missing or defaulted | |
| provenance/claim metadata, and for some packs selection-status blockers. | |
| - Fill the target coverage gaps reported by `tools/validate_eval_target_coverage.py`. | |
| The current coverage config validates, but is intentionally not coverage-ready: | |
| English, Tamang, Magar, Wambule, Bantawa/Rai, Kulung, Yakkha, and Tharu/Rana Tharu | |
| still lack registered real eval packs, and the current registered real-pack recipes | |
| remain non-claim-ready because model commands, GT trust, provenance, calibration, or | |
| row eligibility blockers remain unresolved. A target also remains non-coverage-ready | |
| unless its `eval_status` is exactly `registered_real_pack`; use `partial_real_recognizer_pack`, | |
| `shared_devanagari_pack`, or `missing_real_eval_pack` only when the resulting gap is | |
| intended and visible. | |
| - Generate and wire train/dev leakage manifests for each actual recognizer family and | |
| pass them to scoring runs. The manifest builder exists, but each family still needs | |
| its concrete train/dev label paths recorded in the run recipes, including page-origin | |
| `source_sample_id` for crops selected from full-page eval material. | |
| - Apply native-reader spot-check/calibration to the actual language eval packs, with | |
| enough reviewed rows per script/domain to justify any `machine_converted_spot_checked` | |
| upgrade. The harness now has artifacts and scorer wiring for this, but the reviewed | |
| human evidence still has to be collected. | |
| ## Research Basis | |
| - OCR-D states that OCR quality evaluation requires representative ground truth, | |
| realistic images, manual transcriptions, localization of errors by document/page/ | |
| region, and repeatable workflows: | |
| <https://ocr-d.de/en/spec/ocrd_eval.html> | |
| - The OCR evaluation survey by Neudecker et al. describes CER as edit-distance errors | |
| divided by reference character count and highlights that OCR evaluation depends on | |
| alignment, reading order, layout, and tool-specific normalization choices: | |
| <https://www.primaresearch.org/www/assets/papers/HIP21_CNeudecker_OcrEvalSurvey.pdf> | |
| - Unicode UAX #29 defines extended grapheme clusters as the default programmatic | |
| approximation of user-perceived characters, which is necessary for scripts where one | |
| visible unit can span multiple codepoints: | |
| <https://www.unicode.org/reports/tr29/> | |
| - PaddleOCR's `RecMetric` implementation averages normalized Levenshtein distance per | |
| sample and returns `norm_edit_dis`; this is not the same quantity as micro CER: | |
| <https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/metrics/rec_metric.py> | |
| - Datasheets for Datasets motivates documenting dataset motivation, composition, | |
| collection, and recommended use, which maps directly onto eval-pack provenance: | |
| <https://arxiv.org/abs/1803.09010> | |
| - Model Cards for Model Reporting motivates model documentation with benchmarked | |
| evaluation, intended use, limitations, and evaluation procedure: | |
| <https://arxiv.org/abs/1810.03993> | |
| - Recent OCR evaluation survey work warns that benchmark design can make historical | |
| and marginalized documents invisible when evaluation emphasizes convenient modern | |
| layouts rather than the actual target material: | |
| <https://arxiv.org/html/2603.25761v1> | |