File size: 32,127 Bytes
f9bc0c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 | # Evaluation Harness Goal
Updated: 2026-06-20
## Current Status
This is a parked claim-safety design, not the active product work. The active
2026-06-23 objective is Limbu book OCR: real book pages in, editable Unicode
book text out, with line structure preserved and unresolved text surfaced for
human review. Do not add generic eval-harness features from this document unless
they directly unblock that OCR loop. For the current Limbu work, the useful
evaluation surface is deliberately thin: frozen real-book page acceptance checks,
artifact audits, page/line counts, missed/extra/blank-line rates, review-packet
coverage, and reviewed-only training-pack export validation.
## Goal
The goal is a claim-grade OCR evaluation harness for the Nepal/diaspora recognizer
suite that makes weak publication claims difficult by default, not merely inconvenient:
each language/script result must be scored only from a frozen, versioned eval pack whose
source images, labels, manifest rows, source-document/page/bbox provenance, split role,
domain, ground-truth trust evidence, claim eligibility, canonical pack-content digest,
and script/grapheme coverage are
auditable; each run must record the exact model, checkpoint, config, dictionary,
prediction file, normalization profile, command, timestamp, machine, pack-content digest,
and artifact hashes;
each metric must be recomputed from canonical predictions as corpus micro edit-distance
CER over both Unicode codepoints and UAX #29 extended grapheme clusters, with exact-line
accuracy, matched/unmatched rows, worst examples, confidence intervals, and paired
comparison tests when claims compare systems; recognizer-only oracle-crop scores must
remain separate from page-level OCR scores that include detection, crop lineage, reading
order, and alignment; and the harness must fail or hard-warn on missing images, duplicate
IDs, low match rate, train/dev/test leakage, checkpoint selection on test data, synthetic
or transliterated data presented as real evidence, machine-converted labels presented as
human-verified evidence, human GT without review/agreement evidence, eval packs too small
for the claimed conclusion, uncovered script inventories, and aggregate averages that
hide per-language, per-script, per-domain, or per-GT-status failures. A reviewer opening
one output directory should be able to decide what was scored, against which ground truth,
how trustworthy that ground truth is, whether the test set was frozen and unseen for
selection, what normalization and metric were used, how uncertain the number is, what the
failures look like, what script material was not covered, and whether the result is safe
for a paper claim.
## Requirements
- **Eval packs are versioned evidence objects.** Each pack must include `images/`,
a label file, a manifest JSONL, and a summary/datasheet containing source documents,
page/bbox provenance, language, script, domain, split role, GT source, GT trust
level, known leakage status, claim eligibility, and a `pack_content_sha256` over the
canonical row identity/text/provenance fields so pack drift is detectable across
normalization, scoring, audit, and comparison.
- **Metrics are canonical and micro-averaged.** The harness must compute
total-edit-distance / total-reference-length CER, not per-line macro averages.
It must report both codepoint CER and grapheme-cluster CER, plus exact-line accuracy,
matched/unmatched counts, prediction match rate, total reference units, and
worst-line examples.
- **Point estimates are not enough for claims.** Every claim-facing metric must include
confidence intervals computed by resampling the scoring units, and same-pack system
comparisons must include a paired test over per-row errors. The report must show the
effective sample size per language/script/domain and hard-warn when a pack is too
small for the claimed conclusion.
- **PaddleOCR training metrics are not claim metrics.** PaddleOCR `RecMetric` reports
exact-line `acc` and `norm_edit_dis` as one minus the average per-sample normalized
Levenshtein distance; this may be useful during training, but it is not the
corpus-level micro CER used for claims.
- **Recognition and page OCR are separate tasks.** Oracle-crop recognizer results are
valid recognizer evidence only. Page-level OCR claims require page images, detector
output, crop/reading-order/alignment logic, and a separate end-to-end score.
- **Unicode handling is explicit.** Text is normalized with a named profile, defaulting
to NFC with whitespace preserved. The audit must reject a stale or unsupported
profile that no longer matches the metric recomputation path. Metrics must also
report how many raw GT/reference and prediction strings changed under that profile,
and the audit must recompute those counts from canonical row artifacts so hidden
normalization effects cannot disappear into a single CER. Grapheme metrics use
Unicode extended grapheme clusters; any script-specific tailoring must be recorded.
- **Human verification is first-class metadata.** Machine-converted GT may be scored,
but the output must label it as machine-converted. Paper-grade human-reading claims
require either fully reviewed GT or a documented native-reader calibration sample.
Human-verified GT must record who reviewed it, how much was double-reviewed, and the
observed disagreement or correction rate; a trust label without review evidence is not
claim evidence.
- **Script coverage is evidence, not decoration.** Each eval pack must report the
codepoint, grapheme-cluster, digit, punctuation, conjunct/combining-mark, and
language-specific inventory coverage it exercises, plus gaps against the intended
script inventory. A low CER on a pack that omits common graphemes or conjuncts must not
be eligible for broad script support claims.
- **Reports are inspectable.** Each run should emit `eval-run.json`, `metrics.json`,
`metrics.md`, `predictions.tsv`, source-audited `unmatched.tsv`, `worst-lines.csv`,
`worst-pages.csv`, `sample-review.csv`, `sample-review.html`, `confusions.json`, `command.txt`, and a
human-readable `claim-safety.md` report.
Worst-example rows must carry the row identity, hashes, language/script/domain/GT/split,
claim eligibility, source-document/page/bbox provenance, and reference-artifact trust
fields needed to inspect a failure without reverse-engineering the rest of the run.
- **Reporting is per language/script/domain.** Shared recognizers, such as the
Devanagari model, must still report Majhi, Magar, Sunuwar-Devanagari, Magahi,
Bhojpuri, Thami, Wambule, and Darai separately. Native-script targets such as
Limbu, Sunuwar/Jenticha, Kirat Rai, Lepcha, Tirhuta, Gurung Khema, and Newa must
stay separate from fallback or synthetic-only results.
## Current Implementation Slice
- `tools/run_recognizer_eval.py` scores an existing recognizer prediction file
against a PaddleOCR-style label file. It records non-empty model identity plus config/checkpoint/dictionary,
SHA-256 identities for local checkpoint/config/dictionary artifacts and ties those
identities back to the visible run fields, prediction, label, manifest,
summary/datasheet, script inventory, train/dev manifest, command, machine platform/Python metadata,
normalization, eval-pack version, frozen-pack declaration, selection declaration,
per-row crop image SHA-256 identities and source-document/page/bbox provenance in
`predictions.tsv` and `worst-lines.csv`, deterministic row-
bootstrap 95% confidence intervals for codepoint CER, grapheme CER, and exact-line
accuracy, observed and missing required script codepoint/grapheme inventory coverage,
and claim-safety warnings, and emits a direct `claim-safety.md` report in
the score directory. A claim-safe recognizer run now requires those artifacts to be
readable local files whose hashes can be audited and at least 30 scored rows per
emitted language/script/domain/GT/split slice so tiny or pooled smoke packs cannot
pass as paper evidence. Declared manifest crop-image hashes must
match actual crop bytes; missing script inventory files or missing required
codepoints/graphemes are claim-safety blockers, and rows declared `human_verified`
must carry human review evidence such as reviewer or review-artifact metadata.
- `tools/audit_recognizer_eval_pack.py` audits a recognizer oracle-crop eval pack
before scoring. It verifies label/image/manifest/summary integrity, duplicate
labels, manifest identifiers, and manifest image paths/basenames, manifest text/hash consistency, source-document/
page/bbox coverage, language/script/domain/GT/split metadata, claim eligibility,
frozen-pack declaration, unseen-for-selection declaration, and minimum scoring-unit
counts overall and per language/script/domain/GT/split slice.
- `tools/normalize_recognizer_eval_pack.py` converts existing PaddleOCR-style
recognizer packs into a self-contained canonical pack with copied images, normalized
`eval.txt`, `manifests/recognizer-eval.jsonl`, and `summary.json`. It preserves legacy
provenance fields, fills only explicitly provided defaults, hashes copied images, and
fails on missing images, duplicate labels, duplicate manifest IDs or image
paths/basenames, or manifest/label text mismatches.
- `configs/recognizer-eval-pack-normalization-v1.json` records reproducible
normalization recipes for the current real recognizer packs: Newa, Sunuwar, Lepcha,
Limbu, Kirat Rai, and shared Devanagari. `tools/run_recognizer_pack_normalization.py`
validates those recipes, supports dry-runs, materializes selected packs through the
normalizer, writes per-pack `audit/recognizer-eval-pack-audit.{json,md}` artifacts,
and records pack audit results in `normalization-run.json`.
- `configs/eval-target-coverage-v1.json` records the intended language/script coverage
sets: the broader first-ten document OCR targets and the current real recognizer
packs. `tools/validate_eval_target_coverage.py` compares those targets against the
normalization and eval-run recipe registries so missing packs, missing eval recipes,
non-claim-ready targets, and shared-pack language/script mismatches that could hide
per-language failures are explicit gaps. Target `eval_status` values are constrained
to known states, and only `registered_real_pack` can become coverage-ready; partial,
shared, missing, or invented status labels cannot silently pass as claim support.
- `tools/run_page_ocr_eval.py` scores saved page-level OCR `document.json` outputs
against structured references. It emits an explicit `eval_mode=page_level_ocr`
artifact, separates page OCR from recognizer oracle-crop metrics, records model/run
provenance with local checkpoint/config/dictionary SHA-256 identities, records
scored source image and `document.json` SHA-256 identities plus the embedded
`document.source_path` in `page-metrics.csv`, rejects claim-safe runs whose
`document.source_path` does not match the eval-manifest input, records and audits
the page pack `pack_content_sha256`, recomputes page text micro CER over codepoints
and grapheme clusters,
audits `page-metrics.csv` row identity back to the eval manifest, resolved
`document.json`, scorer metadata defaults, row claim eligibility, and parsed
reference fields, reports
auditable per-language/script/domain/GT/split page slices recomputed from
`page-metrics.csv`, existing document metrics such as reading-order pair accuracy,
table scores, figure scores, reconstruction
metrics, page text exact accuracy, canonical page prediction/reference text
pairs, per-row reference path/hash/trust artifacts, source-document/page/bbox provenance columns, reference coverage,
missing document/reference rows, row-level source-document/page/bbox/
claim-eligibility and language/script/domain/GT metadata coverage, row-level
claim eligibility, line-bbox IoU matching diagnostics when
references provide explicit reading-order bboxes,
worst-page review CSV/HTML artifacts, deterministic page-bootstrap 95% confidence
intervals for page text CER, observed and missing required script codepoint/grapheme
inventory coverage from reference text, and train/dev/eval leakage checks with recorded
train-manifest hashes. Page GT declared as `machine_converted_spot_checked` requires
an eligible native-reader calibration artifact, matching the recognizer-side trust
gate. Page claim-safe runs require at least 30 scored pages per emitted
language/script/domain/GT/split slice, and page score
directories also emit `claim-safety.md` so they can be reviewed without a recipe
wrapper. Duplicate page manifest sample IDs or image paths and stale manifest image
hashes fail before scoring, and missing script inventory files or missing required
codepoints/graphemes are claim-safety blockers. Page rows declared `human_verified`
must also carry human review evidence rather than relying on a trust label alone.
- `tools/run_eval_output_command.py` materializes canonical eval outputs through a
recorded external command before scoring. In `recognizer` mode it writes one
prediction file from a label pack; in `page` mode it runs once per eval-manifest row
and requires each sample to produce `document.json`. It records the exact expanded
command(s), command template, invocation, timestamp, stdout/stderr log paths, output
hashes/sizes, per-page input image paths/hashes, model/checkpoint/config/dictionary metadata, machine details, and
failure details with machine/invocation and label or eval-manifest hashes when a
command does not produce the required artifact. When
`--model-config` is provided without
`--command`, it uses the repo's candidate engine defaults: recognizer mode calls
`tools/predict_recognizer_eval.py`, while page mode calls `ocrtech parse-doc`.
- `tools/predict_recognizer_eval.py` turns a candidate model card into a canonical
recognizer prediction file for oracle-crop eval packs. It reads the label file,
runs the audited `candidate` engine on each crop, writes `image<TAB>prediction<TAB>
confidence`, and records model-card, label, output, machine, and command hashes.
- `tools/prepare_native_spotcheck_bundle.py` creates native-reader review bundles from
recognizer labels, and `tools/apply_native_spotcheck_reviews.py` ingests completed
review CSVs into calibration artifacts. The calibration step validates statuses,
emits reviewed-label and manifest-override artifacts, measures machine-converted label
CER against corrected/native-reviewed text, requires reviewer identity on reviewed
rows, and records whether the spot-check is eligible evidence for treating a pack as
`machine_converted_spot_checked`. Scorers bind that calibration evidence to the
current run's language, script, domain, `machine_converted` source GT status, and
`machine_converted_spot_checked` recommendation. Bundle summaries record source file
hashes, machine metadata, timestamp, and review artifact paths.
- `tools/build_recognizer_leakage_manifest.py` turns recognizer train/dev label files
into JSONL leakage manifests with sample IDs, image hashes, text hashes, and combined
sample hashes. These manifests are the standard input for scorer-side train/dev/eval
leakage checks; text-only mode exists for corpora where image files are intentionally
unavailable. Page-derived crop manifests must also preserve `source_sample_id`, and
the page scorer/audit treat train `source_sample_id` overlap with eval page
`sample_id` as hard leakage.
- `docs/EVAL_HARNESS_RUNBOOK.md` gives the checked operational recipe for chaining
normalization, leakage-manifest generation, output materialization, recognizer scoring,
native-reader calibration, and page-level scoring. It documents the exact claim-safety
flags and the distinction between repo candidate model-card runs and external
PaddleOCR/HF command templates.
- `configs/eval-harness-recipes-v1.json` records concrete current recipe instances for
Newa, Sunuwar, Lepcha, Limbu, Kirat Rai, shared Devanagari, and packaged HF/Paddle
recognizer smoke paths. `tools/validate_eval_harness_recipes.py` verifies that
referenced local paths exist, runs the canonical recognizer eval-pack audit for each
path-complete recognizer pack, statically audits page eval manifests for source-image
hashes, page provenance, row claim eligibility, human-review evidence, and
claim-ready references plus minimum overall and per-slice scoring-unit counts, and
separates structural recipe validity from claim readiness
blockers, including missing, malformed, or recipe-contradicting `summary_file` metadata
for language/script/task/dataset/domain/GT/split/version/eligibility,
missing or malformed `script_inventory_file`, incomplete row provenance,
pack-audit warnings or issues, missing, malformed, ineligible, or mismatched recipe-level
native spot-check calibration when default or audited row GT is `machine_converted_spot_checked`,
scorer-supported normalization profile declarations,
external command templates missing required input/output placeholders or containing
unsupported placeholders, unresolved external prediction command templates, commands still marked as
requiring operator resolution, unhashable checkpoint/config/dictionary artifacts,
malformed or empty `leakage_sources`, declared
checkpoint selection on the test pack, and declared known leakage.
- `tools/run_eval_recipe.py` is the recipe-level recognizer orchestrator. It validates
the selected recipe, builds or wires train/dev leakage manifests, materializes
predictions through the recorded runner, scores the canonical predictions, and writes
`recipe.json`, `recipe-validation.json`, `recipe-run.json`, root-level `audit.json`,
root-level `claim-safety.md`, materialization logs, leakage artifacts, and the scorer
output under one reviewable run directory. The root claim-safety report carries the
score-mode-specific recognizer or page facts from the recipe, score, and audit JSON
files, including script-inventory coverage, metadata contradictions, uncertainty
intervals, leakage status, and the score-mode review artifact list, so it can be
audited for stale reviewer-facing summaries, root `recipe.json` is tied back to the selected
recipe in the hashed registry file, root `recipe-validation.json` is tied back to
`recipe-run.json` and recomputed from the current validator, recipe-run
timestamp/machine/command provenance is checked, root
materialization/audit/claim-safety output pointers must point at the run directory's
own artifacts, root audit summaries must preserve current score-audit issue lists,
warnings, claim-safe status, and score-dir identity, root score summary paths and
hashes must match `score/metrics.json` and `score/eval-run.json`, root train-manifest
path/hash identities must match the score eval-run leakage inputs, the hashed recipe
registry file must sit under the declared recipe root, and the root
prediction/materialization paths, command template, output report, and materialized
model/checkpoint/config/dictionary metadata are tied back to the scored prediction
file hash and score eval-run metadata. Failed recipe attempts write
`recipe-run-error.json` with timestamp, machine, command, recipe validation,
registry hash, root, output directory, exception type, message, and traceback so a
failed run is still attributable.
- `tools/compare_recognizer_eval_runs.py` compares two completed recognizer oracle-crop
score directories on the same frozen eval pack. It pairs canonical `predictions.tsv`
rows by `sample_id`, verifies eval-pack/run identity including `pack_content_sha256`
and row GT identity, emits
`comparison.json`, `comparison.md`, `paired-rows.tsv`, and `command.txt`, reports candidate and
baseline micro CER over codepoints and graphemes, paired wins/losses/ties over
row-level CER, deterministic paired-row bootstrap confidence intervals for the CER
improvement, and an exact two-sided sign-test p-value. Its audit mode checks run
provenance, recomputes the comparison, and rejects stale JSON, Markdown, command,
or paired-row artifacts.
- `tools/compare_page_ocr_eval_runs.py` does the same for completed page-level OCR
score directories, using `page-eval-run.json`, `page-metrics.csv`, and
`text-pairs.tsv`. It verifies page eval-pack identity and page/reference text row
identity including `pack_content_sha256`, emits `comparison.json`,
`comparison.md`, `paired-pages.tsv`, and `command.txt`, reports
paired page text codepoint/grapheme CER improvements, deterministic paired-page
bootstrap intervals, and an exact two-sided sign-test p-value, and can audit
command/provenance plus the comparison artifacts back to the current score
directories.
- `tools/audit_eval_run.py` audits completed recognizer or page score directories, or a
recipe-run root with `score/`, for required artifacts, run metadata, metric arithmetic,
timestamp and command-file consistency, machine platform/Python provenance, recorded
input-file hash consistency, scorer-supported normalization profile consistency,
normalization-change summary recomputation from recognizer `predictions.tsv` or page
`text-pairs.tsv`,
model-artifact hash and run-field consistency,
recognizer `predictions.tsv` source-row recomputation from the recorded
label/prediction/manifest files, row-level
edit-distance and per-language/script/domain/GT/split slice recomputation,
recognizer crop-image and manifest-declared image-hash verification, recognizer source label/prediction
recomputation for label-line, matched-line, unmatched-label, unmatched-prediction,
match-rate metrics, deterministic bootstrap uncertainty intervals, and script
inventory recomputation from recognizer GT rows, recognizer
`unmatched.tsv` identity/hash/score recomputation from the source
label and prediction files, recognizer `confusions.json` recomputation from canonical
predictions, page `reference-artifacts.tsv` recomputation from scored reference
files, page `text-pairs.tsv` recomputation from scored document/reference text,
page `confusions.json` recomputation from those canonical text pairs,
page text CER row and total
recomputation from canonical page text pairs, page row provenance recomputation
from source manifests, scored source-image/
`document.json` hash verification, page bootstrap uncertainty interval recomputation,
script inventory recomputation from page reference text,
and page-metric identity recomputation from manifest/document/reference sources,
run-level language/script/domain/GT/split
list recomputation from canonical row artifacts, page
`line-alignment.csv` row-content and
detection/alignment recomputation from scored document/reference files, including
predicted/reference line IDs, bboxes, text, IoU, and per-line text CER, page
recognizer normalizer/defaulted-metadata count checks against the recorded
summary/datasheet,
direct-run summary/datasheet metadata consistency checks against run language,
script, domain, GT, split, eval-pack version, frozen/unseen, default
claim-eligibility, and task fields,
dictionary coverage recomputation from recognizer GT rows or page reference text,
with GT/reference codepoints outside the recorded dictionary treated as a
claim-safety blocker,
`missing.tsv` recomputation from the eval
manifest, document outputs, and reference files, slice totals, duplicate scored sample/image IDs
in completed output artifacts, ambiguous recognizer manifest image keys,
duplicate page manifest image paths, page manifest
image-hash mismatches, leakage-check presence and leakage-overlap recomputation
from train/eval sources, native spot-check calibration summary checks against the
recorded calibration artifact, metadata-contradiction recomputation from source
provenance signals, provenance/metadata coverage recomputation from source
manifests, page reference-coverage and manifest/scored/missing/claim-eligible count
recomputation from source artifacts, durable
claim-safety blockers
such as metadata contradictions, missing manifests or summary/datasheets, unrun
leakage checks, leakage overlaps, low match rate, unsafe domains, untrusted GT,
missing provenance, missing uncertainty evidence, too-small claim-facing sample size,
missing script inventory evidence, missing required script codepoints/graphemes,
zero-length recognizer GT rows or page text references, recognizer row-level claim
ineligibility, and incomplete page coverage,
warning/claim-safety
consistency, `metrics.md` consistency against `metrics.json`, recomputed
worst-example CSVs, reviewer CSV order/fields, and reviewer HTML contents, direct
score-directory claim reports whose core lines are checked against JSON run facts,
recipe-root claim reports whose recognizer/page score facts are checked against the
nested score directory, and materialization artifacts whose timestamp, machine platform/Python
and invocation metadata, JSON records, model/checkpoint/config/dictionary metadata,
static recipe-level dictionary coverage checks against recognizer GT or page
reference codepoints,
per-record commands/log paths, command-template file,
Markdown report, recognizer prediction path/hash or page document-output directory,
and output hashes are checked against the score
eval-run facts plus claim-report artifacts whose top-level report is also checked
against recipe and score JSON facts.
- The scorer computes corpus micro CER over NFC codepoints and Unicode extended
grapheme clusters, exact-line accuracy, matched/unmatched rows, per-language/script/
domain/GT/split slice metrics, worst lines/pages, review CSV/HTML, and recognizer/page
confusion counts.
- Claim-safety warnings now cover missing manifest or summary, missing eval-pack
version, missing script inventory files, missing required script codepoints or
graphemes, GT/reference codepoints outside the recorded dictionary, unhashable model
artifacts, zero-length recognizer GT rows or page text references, non-frozen or selection-used eval packs, low
prediction match rate, mixed languages/scripts/domains/GT statuses/splits, unsafe
domains, untrusted GT, metadata contradictions such as synthetic/transliterated
provenance declared as real material or machine-converted provenance declared as
`human_verified`, missing source-document/page/bbox/claim-eligibility provenance,
explicit row-level claim ineligibility, missing page row source-document/page/bbox/
claim-eligibility/language/script/domain/GT metadata, normalizer-filled metadata defaults from
canonical pack summaries, missing human-review evidence for `human_verified` rows, missing,
mismatched, or ineligible native spot-check calibration when
`machine_converted_spot_checked` is claimed, known leakage, unrun leakage checks, and
train/dev/eval overlaps from provided JSONL manifests.
- A Newa recognizer artifact was produced at
`outputs/evals/newa-prachalit/newa-native-rescore-2026-06-19-zenodo/`. It is
intentionally **not** claim-safe: GT is machine-converted, the pack is not declared
unseen for selection, page provenance and claim eligibility are missing, and no
train/dev leakage manifest was provided.
- The same Newa eval pack has a standalone audit at
`outputs/evals/newa-prachalit/newa-real-eval-pack-audit-2026-06-19/`. Basic pack
integrity passes (`3180/3180` labels matched to manifest rows and images), but
claim readiness fails because the pack is not declared unseen for selection, has
no page metadata, lacks row-level claim eligibility, and uses machine-converted GT.
## Remaining Gaps
- Resolve remaining remote-only checkpoint/package locations in
`configs/eval-harness-recipes-v1.json` and replace placeholder external commands with
runnable per-family commands. The recipe registry exists and validates local paths,
and `tools/run_eval_recipe.py` can execute resolved recognizer recipes, but several
language-specific PaddleOCR checkpoints are not packaged locally or hash-verifiable,
so the validator must keep them out of claim-ready status.
- Extend page-level evaluation with live detector output capture and crop lineage from
model runners; the current page harness scores saved structured documents and emits
per-line IoU/alignment rows when references provide explicit line bboxes, but it does
not yet run the detector/recognizer pipeline itself.
- Archive and review the materialized canonical eval packs under `outputs/eval-packs/`.
The six current real recognizer packs now exist with per-pack audits, but all six are
structurally non-claim-ready because of machine-converted GT, missing or defaulted
provenance/claim metadata, and for some packs selection-status blockers.
- Fill the target coverage gaps reported by `tools/validate_eval_target_coverage.py`.
The current coverage config validates, but is intentionally not coverage-ready:
English, Tamang, Magar, Wambule, Bantawa/Rai, Kulung, Yakkha, and Tharu/Rana Tharu
still lack registered real eval packs, and the current registered real-pack recipes
remain non-claim-ready because model commands, GT trust, provenance, calibration, or
row eligibility blockers remain unresolved. A target also remains non-coverage-ready
unless its `eval_status` is exactly `registered_real_pack`; use `partial_real_recognizer_pack`,
`shared_devanagari_pack`, or `missing_real_eval_pack` only when the resulting gap is
intended and visible.
- Generate and wire train/dev leakage manifests for each actual recognizer family and
pass them to scoring runs. The manifest builder exists, but each family still needs
its concrete train/dev label paths recorded in the run recipes, including page-origin
`source_sample_id` for crops selected from full-page eval material.
- Apply native-reader spot-check/calibration to the actual language eval packs, with
enough reviewed rows per script/domain to justify any `machine_converted_spot_checked`
upgrade. The harness now has artifacts and scorer wiring for this, but the reviewed
human evidence still has to be collected.
## Research Basis
- OCR-D states that OCR quality evaluation requires representative ground truth,
realistic images, manual transcriptions, localization of errors by document/page/
region, and repeatable workflows:
<https://ocr-d.de/en/spec/ocrd_eval.html>
- The OCR evaluation survey by Neudecker et al. describes CER as edit-distance errors
divided by reference character count and highlights that OCR evaluation depends on
alignment, reading order, layout, and tool-specific normalization choices:
<https://www.primaresearch.org/www/assets/papers/HIP21_CNeudecker_OcrEvalSurvey.pdf>
- Unicode UAX #29 defines extended grapheme clusters as the default programmatic
approximation of user-perceived characters, which is necessary for scripts where one
visible unit can span multiple codepoints:
<https://www.unicode.org/reports/tr29/>
- PaddleOCR's `RecMetric` implementation averages normalized Levenshtein distance per
sample and returns `norm_edit_dis`; this is not the same quantity as micro CER:
<https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/metrics/rec_metric.py>
- Datasheets for Datasets motivates documenting dataset motivation, composition,
collection, and recommended use, which maps directly onto eval-pack provenance:
<https://arxiv.org/abs/1803.09010>
- Model Cards for Model Reporting motivates model documentation with benchmarked
evaluation, intended use, limitations, and evaluation procedure:
<https://arxiv.org/abs/1810.03993>
- Recent OCR evaluation survey work warns that benchmark design can make historical
and marginalized documents invisible when evaluation emphasizes convenient modern
layouts rather than the actual target material:
<https://arxiv.org/html/2603.25761v1>
|