File size: 32,127 Bytes
f9bc0c1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
# Evaluation Harness Goal

Updated: 2026-06-20

## Current Status

This is a parked claim-safety design, not the active product work. The active
2026-06-23 objective is Limbu book OCR: real book pages in, editable Unicode
book text out, with line structure preserved and unresolved text surfaced for
human review. Do not add generic eval-harness features from this document unless
they directly unblock that OCR loop. For the current Limbu work, the useful
evaluation surface is deliberately thin: frozen real-book page acceptance checks,
artifact audits, page/line counts, missed/extra/blank-line rates, review-packet
coverage, and reviewed-only training-pack export validation.

## Goal

The goal is a claim-grade OCR evaluation harness for the Nepal/diaspora recognizer
suite that makes weak publication claims difficult by default, not merely inconvenient:
each language/script result must be scored only from a frozen, versioned eval pack whose
source images, labels, manifest rows, source-document/page/bbox provenance, split role,
domain, ground-truth trust evidence, claim eligibility, canonical pack-content digest,
and script/grapheme coverage are
auditable; each run must record the exact model, checkpoint, config, dictionary,
prediction file, normalization profile, command, timestamp, machine, pack-content digest,
and artifact hashes;
each metric must be recomputed from canonical predictions as corpus micro edit-distance
CER over both Unicode codepoints and UAX #29 extended grapheme clusters, with exact-line
accuracy, matched/unmatched rows, worst examples, confidence intervals, and paired
comparison tests when claims compare systems; recognizer-only oracle-crop scores must
remain separate from page-level OCR scores that include detection, crop lineage, reading
order, and alignment; and the harness must fail or hard-warn on missing images, duplicate
IDs, low match rate, train/dev/test leakage, checkpoint selection on test data, synthetic
or transliterated data presented as real evidence, machine-converted labels presented as
human-verified evidence, human GT without review/agreement evidence, eval packs too small
for the claimed conclusion, uncovered script inventories, and aggregate averages that
hide per-language, per-script, per-domain, or per-GT-status failures. A reviewer opening
one output directory should be able to decide what was scored, against which ground truth,
how trustworthy that ground truth is, whether the test set was frozen and unseen for
selection, what normalization and metric were used, how uncertain the number is, what the
failures look like, what script material was not covered, and whether the result is safe
for a paper claim.

## Requirements

- **Eval packs are versioned evidence objects.** Each pack must include `images/`,
  a label file, a manifest JSONL, and a summary/datasheet containing source documents,
  page/bbox provenance, language, script, domain, split role, GT source, GT trust
  level, known leakage status, claim eligibility, and a `pack_content_sha256` over the
  canonical row identity/text/provenance fields so pack drift is detectable across
  normalization, scoring, audit, and comparison.
- **Metrics are canonical and micro-averaged.** The harness must compute
  total-edit-distance / total-reference-length CER, not per-line macro averages.
  It must report both codepoint CER and grapheme-cluster CER, plus exact-line accuracy,
  matched/unmatched counts, prediction match rate, total reference units, and
  worst-line examples.
- **Point estimates are not enough for claims.** Every claim-facing metric must include
  confidence intervals computed by resampling the scoring units, and same-pack system
  comparisons must include a paired test over per-row errors. The report must show the
  effective sample size per language/script/domain and hard-warn when a pack is too
  small for the claimed conclusion.
- **PaddleOCR training metrics are not claim metrics.** PaddleOCR `RecMetric` reports
  exact-line `acc` and `norm_edit_dis` as one minus the average per-sample normalized
  Levenshtein distance; this may be useful during training, but it is not the
  corpus-level micro CER used for claims.
- **Recognition and page OCR are separate tasks.** Oracle-crop recognizer results are
  valid recognizer evidence only. Page-level OCR claims require page images, detector
  output, crop/reading-order/alignment logic, and a separate end-to-end score.
- **Unicode handling is explicit.** Text is normalized with a named profile, defaulting
  to NFC with whitespace preserved. The audit must reject a stale or unsupported
  profile that no longer matches the metric recomputation path. Metrics must also
  report how many raw GT/reference and prediction strings changed under that profile,
  and the audit must recompute those counts from canonical row artifacts so hidden
  normalization effects cannot disappear into a single CER. Grapheme metrics use
  Unicode extended grapheme clusters; any script-specific tailoring must be recorded.
- **Human verification is first-class metadata.** Machine-converted GT may be scored,
  but the output must label it as machine-converted. Paper-grade human-reading claims
  require either fully reviewed GT or a documented native-reader calibration sample.
  Human-verified GT must record who reviewed it, how much was double-reviewed, and the
  observed disagreement or correction rate; a trust label without review evidence is not
  claim evidence.
- **Script coverage is evidence, not decoration.** Each eval pack must report the
  codepoint, grapheme-cluster, digit, punctuation, conjunct/combining-mark, and
  language-specific inventory coverage it exercises, plus gaps against the intended
  script inventory. A low CER on a pack that omits common graphemes or conjuncts must not
  be eligible for broad script support claims.
- **Reports are inspectable.** Each run should emit `eval-run.json`, `metrics.json`,
  `metrics.md`, `predictions.tsv`, source-audited `unmatched.tsv`, `worst-lines.csv`,
  `worst-pages.csv`, `sample-review.csv`, `sample-review.html`, `confusions.json`, `command.txt`, and a
  human-readable `claim-safety.md` report.
  Worst-example rows must carry the row identity, hashes, language/script/domain/GT/split,
  claim eligibility, source-document/page/bbox provenance, and reference-artifact trust
  fields needed to inspect a failure without reverse-engineering the rest of the run.
- **Reporting is per language/script/domain.** Shared recognizers, such as the
  Devanagari model, must still report Majhi, Magar, Sunuwar-Devanagari, Magahi,
  Bhojpuri, Thami, Wambule, and Darai separately. Native-script targets such as
  Limbu, Sunuwar/Jenticha, Kirat Rai, Lepcha, Tirhuta, Gurung Khema, and Newa must
  stay separate from fallback or synthetic-only results.

## Current Implementation Slice

- `tools/run_recognizer_eval.py` scores an existing recognizer prediction file
  against a PaddleOCR-style label file. It records non-empty model identity plus config/checkpoint/dictionary,
  SHA-256 identities for local checkpoint/config/dictionary artifacts and ties those
  identities back to the visible run fields, prediction, label, manifest,
  summary/datasheet, script inventory, train/dev manifest, command, machine platform/Python metadata,
  normalization, eval-pack version, frozen-pack declaration, selection declaration,
  per-row crop image SHA-256 identities and source-document/page/bbox provenance in
  `predictions.tsv` and `worst-lines.csv`, deterministic row-
  bootstrap 95% confidence intervals for codepoint CER, grapheme CER, and exact-line
  accuracy, observed and missing required script codepoint/grapheme inventory coverage,
  and claim-safety warnings, and emits a direct `claim-safety.md` report in
  the score directory. A claim-safe recognizer run now requires those artifacts to be
  readable local files whose hashes can be audited and at least 30 scored rows per
  emitted language/script/domain/GT/split slice so tiny or pooled smoke packs cannot
  pass as paper evidence. Declared manifest crop-image hashes must
  match actual crop bytes; missing script inventory files or missing required
  codepoints/graphemes are claim-safety blockers, and rows declared `human_verified`
  must carry human review evidence such as reviewer or review-artifact metadata.
- `tools/audit_recognizer_eval_pack.py` audits a recognizer oracle-crop eval pack
  before scoring. It verifies label/image/manifest/summary integrity, duplicate
  labels, manifest identifiers, and manifest image paths/basenames, manifest text/hash consistency, source-document/
  page/bbox coverage, language/script/domain/GT/split metadata, claim eligibility,
  frozen-pack declaration, unseen-for-selection declaration, and minimum scoring-unit
  counts overall and per language/script/domain/GT/split slice.
- `tools/normalize_recognizer_eval_pack.py` converts existing PaddleOCR-style
  recognizer packs into a self-contained canonical pack with copied images, normalized
  `eval.txt`, `manifests/recognizer-eval.jsonl`, and `summary.json`. It preserves legacy
  provenance fields, fills only explicitly provided defaults, hashes copied images, and
  fails on missing images, duplicate labels, duplicate manifest IDs or image
  paths/basenames, or manifest/label text mismatches.
- `configs/recognizer-eval-pack-normalization-v1.json` records reproducible
  normalization recipes for the current real recognizer packs: Newa, Sunuwar, Lepcha,
  Limbu, Kirat Rai, and shared Devanagari. `tools/run_recognizer_pack_normalization.py`
  validates those recipes, supports dry-runs, materializes selected packs through the
  normalizer, writes per-pack `audit/recognizer-eval-pack-audit.{json,md}` artifacts,
  and records pack audit results in `normalization-run.json`.
- `configs/eval-target-coverage-v1.json` records the intended language/script coverage
  sets: the broader first-ten document OCR targets and the current real recognizer
  packs. `tools/validate_eval_target_coverage.py` compares those targets against the
  normalization and eval-run recipe registries so missing packs, missing eval recipes,
  non-claim-ready targets, and shared-pack language/script mismatches that could hide
  per-language failures are explicit gaps. Target `eval_status` values are constrained
  to known states, and only `registered_real_pack` can become coverage-ready; partial,
  shared, missing, or invented status labels cannot silently pass as claim support.
- `tools/run_page_ocr_eval.py` scores saved page-level OCR `document.json` outputs
  against structured references. It emits an explicit `eval_mode=page_level_ocr`
  artifact, separates page OCR from recognizer oracle-crop metrics, records model/run
  provenance with local checkpoint/config/dictionary SHA-256 identities, records
  scored source image and `document.json` SHA-256 identities plus the embedded
  `document.source_path` in `page-metrics.csv`, rejects claim-safe runs whose
  `document.source_path` does not match the eval-manifest input, records and audits
  the page pack `pack_content_sha256`, recomputes page text micro CER over codepoints
  and grapheme clusters,
  audits `page-metrics.csv` row identity back to the eval manifest, resolved
  `document.json`, scorer metadata defaults, row claim eligibility, and parsed
  reference fields, reports
  auditable per-language/script/domain/GT/split page slices recomputed from
  `page-metrics.csv`, existing document metrics such as reading-order pair accuracy,
  table scores, figure scores, reconstruction
  metrics, page text exact accuracy, canonical page prediction/reference text
  pairs, per-row reference path/hash/trust artifacts, source-document/page/bbox provenance columns, reference coverage,
  missing document/reference rows, row-level source-document/page/bbox/
  claim-eligibility and language/script/domain/GT metadata coverage, row-level
  claim eligibility, line-bbox IoU matching diagnostics when
  references provide explicit reading-order bboxes,
  worst-page review CSV/HTML artifacts, deterministic page-bootstrap 95% confidence
  intervals for page text CER, observed and missing required script codepoint/grapheme
  inventory coverage from reference text, and train/dev/eval leakage checks with recorded
  train-manifest hashes. Page GT declared as `machine_converted_spot_checked` requires
  an eligible native-reader calibration artifact, matching the recognizer-side trust
  gate. Page claim-safe runs require at least 30 scored pages per emitted
  language/script/domain/GT/split slice, and page score
  directories also emit `claim-safety.md` so they can be reviewed without a recipe
  wrapper. Duplicate page manifest sample IDs or image paths and stale manifest image
  hashes fail before scoring, and missing script inventory files or missing required
  codepoints/graphemes are claim-safety blockers. Page rows declared `human_verified`
  must also carry human review evidence rather than relying on a trust label alone.
- `tools/run_eval_output_command.py` materializes canonical eval outputs through a
  recorded external command before scoring. In `recognizer` mode it writes one
  prediction file from a label pack; in `page` mode it runs once per eval-manifest row
  and requires each sample to produce `document.json`. It records the exact expanded
  command(s), command template, invocation, timestamp, stdout/stderr log paths, output
  hashes/sizes, per-page input image paths/hashes, model/checkpoint/config/dictionary metadata, machine details, and
  failure details with machine/invocation and label or eval-manifest hashes when a
  command does not produce the required artifact. When
  `--model-config` is provided without
  `--command`, it uses the repo's candidate engine defaults: recognizer mode calls
  `tools/predict_recognizer_eval.py`, while page mode calls `ocrtech parse-doc`.
- `tools/predict_recognizer_eval.py` turns a candidate model card into a canonical
  recognizer prediction file for oracle-crop eval packs. It reads the label file,
  runs the audited `candidate` engine on each crop, writes `image<TAB>prediction<TAB>
  confidence`, and records model-card, label, output, machine, and command hashes.
- `tools/prepare_native_spotcheck_bundle.py` creates native-reader review bundles from
  recognizer labels, and `tools/apply_native_spotcheck_reviews.py` ingests completed
  review CSVs into calibration artifacts. The calibration step validates statuses,
  emits reviewed-label and manifest-override artifacts, measures machine-converted label
  CER against corrected/native-reviewed text, requires reviewer identity on reviewed
  rows, and records whether the spot-check is eligible evidence for treating a pack as
  `machine_converted_spot_checked`. Scorers bind that calibration evidence to the
  current run's language, script, domain, `machine_converted` source GT status, and
  `machine_converted_spot_checked` recommendation. Bundle summaries record source file
  hashes, machine metadata, timestamp, and review artifact paths.
- `tools/build_recognizer_leakage_manifest.py` turns recognizer train/dev label files
  into JSONL leakage manifests with sample IDs, image hashes, text hashes, and combined
  sample hashes. These manifests are the standard input for scorer-side train/dev/eval
  leakage checks; text-only mode exists for corpora where image files are intentionally
  unavailable. Page-derived crop manifests must also preserve `source_sample_id`, and
  the page scorer/audit treat train `source_sample_id` overlap with eval page
  `sample_id` as hard leakage.
- `docs/EVAL_HARNESS_RUNBOOK.md` gives the checked operational recipe for chaining
  normalization, leakage-manifest generation, output materialization, recognizer scoring,
  native-reader calibration, and page-level scoring. It documents the exact claim-safety
  flags and the distinction between repo candidate model-card runs and external
  PaddleOCR/HF command templates.
- `configs/eval-harness-recipes-v1.json` records concrete current recipe instances for
  Newa, Sunuwar, Lepcha, Limbu, Kirat Rai, shared Devanagari, and packaged HF/Paddle
  recognizer smoke paths. `tools/validate_eval_harness_recipes.py` verifies that
  referenced local paths exist, runs the canonical recognizer eval-pack audit for each
  path-complete recognizer pack, statically audits page eval manifests for source-image
  hashes, page provenance, row claim eligibility, human-review evidence, and
  claim-ready references plus minimum overall and per-slice scoring-unit counts, and
  separates structural recipe validity from claim readiness
  blockers, including missing, malformed, or recipe-contradicting `summary_file` metadata
  for language/script/task/dataset/domain/GT/split/version/eligibility,
  missing or malformed `script_inventory_file`, incomplete row provenance,
  pack-audit warnings or issues, missing, malformed, ineligible, or mismatched recipe-level
  native spot-check calibration when default or audited row GT is `machine_converted_spot_checked`,
  scorer-supported normalization profile declarations,
  external command templates missing required input/output placeholders or containing
  unsupported placeholders, unresolved external prediction command templates, commands still marked as
  requiring operator resolution, unhashable checkpoint/config/dictionary artifacts,
  malformed or empty `leakage_sources`, declared
  checkpoint selection on the test pack, and declared known leakage.
- `tools/run_eval_recipe.py` is the recipe-level recognizer orchestrator. It validates
  the selected recipe, builds or wires train/dev leakage manifests, materializes
  predictions through the recorded runner, scores the canonical predictions, and writes
  `recipe.json`, `recipe-validation.json`, `recipe-run.json`, root-level `audit.json`,
  root-level `claim-safety.md`, materialization logs, leakage artifacts, and the scorer
  output under one reviewable run directory. The root claim-safety report carries the
  score-mode-specific recognizer or page facts from the recipe, score, and audit JSON
  files, including script-inventory coverage, metadata contradictions, uncertainty
  intervals, leakage status, and the score-mode review artifact list, so it can be
  audited for stale reviewer-facing summaries, root `recipe.json` is tied back to the selected
  recipe in the hashed registry file, root `recipe-validation.json` is tied back to
  `recipe-run.json` and recomputed from the current validator, recipe-run
  timestamp/machine/command provenance is checked, root
  materialization/audit/claim-safety output pointers must point at the run directory's
  own artifacts, root audit summaries must preserve current score-audit issue lists,
  warnings, claim-safe status, and score-dir identity, root score summary paths and
  hashes must match `score/metrics.json` and `score/eval-run.json`, root train-manifest
  path/hash identities must match the score eval-run leakage inputs, the hashed recipe
  registry file must sit under the declared recipe root, and the root
  prediction/materialization paths, command template, output report, and materialized
  model/checkpoint/config/dictionary metadata are tied back to the scored prediction
  file hash and score eval-run metadata. Failed recipe attempts write
  `recipe-run-error.json` with timestamp, machine, command, recipe validation,
  registry hash, root, output directory, exception type, message, and traceback so a
  failed run is still attributable.
- `tools/compare_recognizer_eval_runs.py` compares two completed recognizer oracle-crop
  score directories on the same frozen eval pack. It pairs canonical `predictions.tsv`
  rows by `sample_id`, verifies eval-pack/run identity including `pack_content_sha256`
  and row GT identity, emits
  `comparison.json`, `comparison.md`, `paired-rows.tsv`, and `command.txt`, reports candidate and
  baseline micro CER over codepoints and graphemes, paired wins/losses/ties over
  row-level CER, deterministic paired-row bootstrap confidence intervals for the CER
  improvement, and an exact two-sided sign-test p-value. Its audit mode checks run
  provenance, recomputes the comparison, and rejects stale JSON, Markdown, command,
  or paired-row artifacts.
- `tools/compare_page_ocr_eval_runs.py` does the same for completed page-level OCR
  score directories, using `page-eval-run.json`, `page-metrics.csv`, and
  `text-pairs.tsv`. It verifies page eval-pack identity and page/reference text row
  identity including `pack_content_sha256`, emits `comparison.json`,
  `comparison.md`, `paired-pages.tsv`, and `command.txt`, reports
  paired page text codepoint/grapheme CER improvements, deterministic paired-page
  bootstrap intervals, and an exact two-sided sign-test p-value, and can audit
  command/provenance plus the comparison artifacts back to the current score
  directories.
- `tools/audit_eval_run.py` audits completed recognizer or page score directories, or a
  recipe-run root with `score/`, for required artifacts, run metadata, metric arithmetic,
  timestamp and command-file consistency, machine platform/Python provenance, recorded
  input-file hash consistency, scorer-supported normalization profile consistency,
  normalization-change summary recomputation from recognizer `predictions.tsv` or page
  `text-pairs.tsv`,
  model-artifact hash and run-field consistency,
  recognizer `predictions.tsv` source-row recomputation from the recorded
  label/prediction/manifest files, row-level
  edit-distance and per-language/script/domain/GT/split slice recomputation,
  recognizer crop-image and manifest-declared image-hash verification, recognizer source label/prediction
  recomputation for label-line, matched-line, unmatched-label, unmatched-prediction,
  match-rate metrics, deterministic bootstrap uncertainty intervals, and script
  inventory recomputation from recognizer GT rows, recognizer
  `unmatched.tsv` identity/hash/score recomputation from the source
  label and prediction files, recognizer `confusions.json` recomputation from canonical
  predictions, page `reference-artifacts.tsv` recomputation from scored reference
  files, page `text-pairs.tsv` recomputation from scored document/reference text,
  page `confusions.json` recomputation from those canonical text pairs,
  page text CER row and total
  recomputation from canonical page text pairs, page row provenance recomputation
  from source manifests, scored source-image/
  `document.json` hash verification, page bootstrap uncertainty interval recomputation,
  script inventory recomputation from page reference text,
  and page-metric identity recomputation from manifest/document/reference sources,
  run-level language/script/domain/GT/split
  list recomputation from canonical row artifacts, page
  `line-alignment.csv` row-content and
  detection/alignment recomputation from scored document/reference files, including
  predicted/reference line IDs, bboxes, text, IoU, and per-line text CER, page
  recognizer normalizer/defaulted-metadata count checks against the recorded
  summary/datasheet,
  direct-run summary/datasheet metadata consistency checks against run language,
  script, domain, GT, split, eval-pack version, frozen/unseen, default
  claim-eligibility, and task fields,
  dictionary coverage recomputation from recognizer GT rows or page reference text,
  with GT/reference codepoints outside the recorded dictionary treated as a
  claim-safety blocker,
  `missing.tsv` recomputation from the eval
  manifest, document outputs, and reference files, slice totals, duplicate scored sample/image IDs
  in completed output artifacts, ambiguous recognizer manifest image keys,
  duplicate page manifest image paths, page manifest
  image-hash mismatches, leakage-check presence and leakage-overlap recomputation
  from train/eval sources, native spot-check calibration summary checks against the
  recorded calibration artifact, metadata-contradiction recomputation from source
  provenance signals, provenance/metadata coverage recomputation from source
  manifests, page reference-coverage and manifest/scored/missing/claim-eligible count
  recomputation from source artifacts, durable
  claim-safety blockers
  such as metadata contradictions, missing manifests or summary/datasheets, unrun
  leakage checks, leakage overlaps, low match rate, unsafe domains, untrusted GT,
  missing provenance, missing uncertainty evidence, too-small claim-facing sample size,
  missing script inventory evidence, missing required script codepoints/graphemes,
  zero-length recognizer GT rows or page text references, recognizer row-level claim
  ineligibility, and incomplete page coverage,
  warning/claim-safety
  consistency, `metrics.md` consistency against `metrics.json`, recomputed
  worst-example CSVs, reviewer CSV order/fields, and reviewer HTML contents, direct
  score-directory claim reports whose core lines are checked against JSON run facts,
  recipe-root claim reports whose recognizer/page score facts are checked against the
  nested score directory, and materialization artifacts whose timestamp, machine platform/Python
  and invocation metadata, JSON records, model/checkpoint/config/dictionary metadata,
  static recipe-level dictionary coverage checks against recognizer GT or page
  reference codepoints,
  per-record commands/log paths, command-template file,
  Markdown report, recognizer prediction path/hash or page document-output directory,
  and output hashes are checked against the score
  eval-run facts plus claim-report artifacts whose top-level report is also checked
  against recipe and score JSON facts.
- The scorer computes corpus micro CER over NFC codepoints and Unicode extended
  grapheme clusters, exact-line accuracy, matched/unmatched rows, per-language/script/
  domain/GT/split slice metrics, worst lines/pages, review CSV/HTML, and recognizer/page
  confusion counts.
- Claim-safety warnings now cover missing manifest or summary, missing eval-pack
  version, missing script inventory files, missing required script codepoints or
  graphemes, GT/reference codepoints outside the recorded dictionary, unhashable model
  artifacts, zero-length recognizer GT rows or page text references, non-frozen or selection-used eval packs, low
  prediction match rate, mixed languages/scripts/domains/GT statuses/splits, unsafe
  domains, untrusted GT, metadata contradictions such as synthetic/transliterated
  provenance declared as real material or machine-converted provenance declared as
  `human_verified`, missing source-document/page/bbox/claim-eligibility provenance,
  explicit row-level claim ineligibility, missing page row source-document/page/bbox/
  claim-eligibility/language/script/domain/GT metadata, normalizer-filled metadata defaults from
  canonical pack summaries, missing human-review evidence for `human_verified` rows, missing,
  mismatched, or ineligible native spot-check calibration when
  `machine_converted_spot_checked` is claimed, known leakage, unrun leakage checks, and
  train/dev/eval overlaps from provided JSONL manifests.
- A Newa recognizer artifact was produced at
  `outputs/evals/newa-prachalit/newa-native-rescore-2026-06-19-zenodo/`. It is
  intentionally **not** claim-safe: GT is machine-converted, the pack is not declared
  unseen for selection, page provenance and claim eligibility are missing, and no
  train/dev leakage manifest was provided.
- The same Newa eval pack has a standalone audit at
  `outputs/evals/newa-prachalit/newa-real-eval-pack-audit-2026-06-19/`. Basic pack
  integrity passes (`3180/3180` labels matched to manifest rows and images), but
  claim readiness fails because the pack is not declared unseen for selection, has
  no page metadata, lacks row-level claim eligibility, and uses machine-converted GT.

## Remaining Gaps

- Resolve remaining remote-only checkpoint/package locations in
  `configs/eval-harness-recipes-v1.json` and replace placeholder external commands with
  runnable per-family commands. The recipe registry exists and validates local paths,
  and `tools/run_eval_recipe.py` can execute resolved recognizer recipes, but several
  language-specific PaddleOCR checkpoints are not packaged locally or hash-verifiable,
  so the validator must keep them out of claim-ready status.
- Extend page-level evaluation with live detector output capture and crop lineage from
  model runners; the current page harness scores saved structured documents and emits
  per-line IoU/alignment rows when references provide explicit line bboxes, but it does
  not yet run the detector/recognizer pipeline itself.
- Archive and review the materialized canonical eval packs under `outputs/eval-packs/`.
  The six current real recognizer packs now exist with per-pack audits, but all six are
  structurally non-claim-ready because of machine-converted GT, missing or defaulted
  provenance/claim metadata, and for some packs selection-status blockers.
- Fill the target coverage gaps reported by `tools/validate_eval_target_coverage.py`.
  The current coverage config validates, but is intentionally not coverage-ready:
  English, Tamang, Magar, Wambule, Bantawa/Rai, Kulung, Yakkha, and Tharu/Rana Tharu
  still lack registered real eval packs, and the current registered real-pack recipes
  remain non-claim-ready because model commands, GT trust, provenance, calibration, or
  row eligibility blockers remain unresolved. A target also remains non-coverage-ready
  unless its `eval_status` is exactly `registered_real_pack`; use `partial_real_recognizer_pack`,
  `shared_devanagari_pack`, or `missing_real_eval_pack` only when the resulting gap is
  intended and visible.
- Generate and wire train/dev leakage manifests for each actual recognizer family and
  pass them to scoring runs. The manifest builder exists, but each family still needs
  its concrete train/dev label paths recorded in the run recipes, including page-origin
  `source_sample_id` for crops selected from full-page eval material.
- Apply native-reader spot-check/calibration to the actual language eval packs, with
  enough reviewed rows per script/domain to justify any `machine_converted_spot_checked`
  upgrade. The harness now has artifacts and scorer wiring for this, but the reviewed
  human evidence still has to be collected.

## Research Basis

- OCR-D states that OCR quality evaluation requires representative ground truth,
  realistic images, manual transcriptions, localization of errors by document/page/
  region, and repeatable workflows:
  <https://ocr-d.de/en/spec/ocrd_eval.html>
- The OCR evaluation survey by Neudecker et al. describes CER as edit-distance errors
  divided by reference character count and highlights that OCR evaluation depends on
  alignment, reading order, layout, and tool-specific normalization choices:
  <https://www.primaresearch.org/www/assets/papers/HIP21_CNeudecker_OcrEvalSurvey.pdf>
- Unicode UAX #29 defines extended grapheme clusters as the default programmatic
  approximation of user-perceived characters, which is necessary for scripts where one
  visible unit can span multiple codepoints:
  <https://www.unicode.org/reports/tr29/>
- PaddleOCR's `RecMetric` implementation averages normalized Levenshtein distance per
  sample and returns `norm_edit_dis`; this is not the same quantity as micro CER:
  <https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/metrics/rec_metric.py>
- Datasheets for Datasets motivates documenting dataset motivation, composition,
  collection, and recommended use, which maps directly onto eval-pack provenance:
  <https://arxiv.org/abs/1803.09010>
- Model Cards for Model Reporting motivates model documentation with benchmarked
  evaluation, intended use, limitations, and evaluation procedure:
  <https://arxiv.org/abs/1810.03993>
- Recent OCR evaluation survey work warns that benchmark design can make historical
  and marginalized documents invisible when evaluation emphasizes convenient modern
  layouts rather than the actual target material:
  <https://arxiv.org/html/2603.25761v1>