File size: 16,897 Bytes
f9bc0c1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
# Claim Review

`review-claim` is the portfolio-level gate for a SOTA claim. `validate-claim` can tell you whether one experiment passed its configured gates. It cannot tell you whether you have enough evidence across the internal Nepali/English pack and public suites to say "better than what is out there." That is what `review-claim` is for.

## What It Audits

`review-claim` checks that:

- every required experiment slot is backed by a real `experiment.json`
- every referenced eval manifest and benchmark report still exists at the
  recorded path, and the eval manifest matches the experiment's recorded
  SHA-256
- successful benchmark rows are bound back to sample ids and input paths in the
  reviewed eval manifest, and row slice labels must match the manifest entry
- the expected datasets or manifest patterns are present
- required baselines were actually run
- required metric families are present on successful rows for the required baselines
- required metric families are present on the slices they are supposed to
  substantiate when `required_metric_slices` is configured
- benchmark reports do not contain error rows when the slot requires clean runs;
  status counts are derived from `benchmark/report.json`, not trusted from the
  copied `experiment.json` summary
- required validation status is present for internal claim-grade experiments,
  and the referenced `validation.json` agrees with `experiment.json`
- required validation gates passed for the baseline/metric/slice comparisons
  declared by `required_validation_gates`
- when configured, candidate fallback diagnostics are present and the aggregate
  fallback trigger/failure rates stay under the slot limits
- required slices exist for the internal Nepali/English pack
- candidate experiments all point to the same candidate model config
- the candidate model card matches experiment hashes and passes `audit-model`,
  with or without an explicit `--model-config`
- if candidate experiments record a fallback model card, that card exists,
  matches the experiment SHA-256, and passes `audit-model`
- when configured, the candidate model card itself records the same fallback
  component in `provenance.fallback_model_card`
- when configured, the candidate model card proves it includes a trained local
  recognizer with artifacts, train/eval manifest hashes, and training-summary
  provenance
- when configured, the candidate model card references a passing admission
  validation report before it can enter portfolio review
- routed candidates such as `quality_select_composite`, `quality_ranked_ensemble`, and `script_select_composite` were calibrated on a separate eval manifest, not on one of the reviewed experiment manifests
- routed candidates also fail review if their selected calibration trial collapses to one engine on every calibration sample
- routed candidates also fail review if a reviewed benchmark shows they are exactly indistinguishable from one of their component baselines on paired holdout CER

Metric coverage is a hard gate because the target claim is a document-understanding
claim, not a plain OCR claim. A required experiment can now declare
`required_metrics`; each metric must appear as a finite value on at least one
successful row for every required baseline in that slot. For the internal
Nepali/English document pack, the SOTA config requires text OCR metrics
(`cer`, `wer`), document reconstruction
(`document_reconstruction_similarity`), table reconstruction
(`table_cell_f1`), reading order (`reading_order_pair_accuracy`), and figure/caption metrics
(`figure_detection_f1`, `figure_caption_cer`). Public table/document slots
declare their own metric set.

For claim-grade slots, also declare `required_metric_slices`. This prevents a
metric from satisfying the portfolio check on the wrong document type. For
example, `table_cell_f1` must be finite on a `table` or document-parsing row,
and `figure_caption_cer` must be finite on a `figure` row.

Metric presence is not enough. When a required slot includes `candidate` and
one or more external baselines, `review-claim` also requires paired metric
coverage on the same sample ids. If `required_metric_slices` is configured, the
pairing is checked on those slices; otherwise it is checked across all rows.
This prevents a SOTA review from passing because candidate metrics and baseline
metrics both exist somewhere in the report but never compare the same documents.

Sample coverage is also a hard gate. Every `require_baselines` entry must have a
successful benchmark row for every sample in the reviewed eval manifest. This
prevents a report from pointing at a hard eval pack while only running an easy
subset.

When a slot declares `require_validation_status`, `review-claim` reads the
referenced validation report and verifies `claim_status`. For `validated`, it
also requires `passed: true`. The copied status inside `experiment.json` is not
enough on its own. The validation artifact must also carry provenance linking it
to the same benchmark report, eval manifest, validation config, model cards,
and train manifest hashes referenced by the experiment.

Use `required_validation_gates` for claim-grade slots. Metric coverage proves
that a benchmark emitted a metric; passing validation gates prove the candidate
beat a named baseline under the configured sample count, slices, margins, and
statistical checks. Review requires each matched gate to carry quantitative
paired evidence, not just `passed: true`: positive `pairs`, finite candidate
and baseline means, finite absolute improvement, and win/loss/tie counts that
sum to the pair count. It also re-checks the threshold fields embedded in the
gate result, including `min_pairs`, margins, `min_pair_win_rate`, `max_p_value`,
and `require_ci_above_zero`, so a hand-edited report cannot mark a weak gate as
passed without satisfying its own configured test.

The SOTA validation configs also use `min_pair_win_rate` on improvement gates.
This is checked as wins divided by all paired samples, not only non-tied pairs.
It prevents an average-only improvement, or a single win with many ties, from
supporting a broad "better OCR/document engine" claim.

`required_validation_gates` is only valid on slots that also declare
`require_validation_status`. Each required gate baseline must also appear in
`require_baselines`, and each gate metric must appear in `required_metrics`.

For claim-grade slots, set `required_validation_config` to the checked-in
validation config path. `review-claim` hashes that file and requires
`experiment.json` to record the same `validation_config_sha256`. This prevents a
portfolio from passing with an easier local validation config that happens to
contain a subset of the expected gates.

For hybrid candidates, use `require_candidate_fallback_metrics`,
`max_candidate_fallback_trigger_rate`, and
`max_candidate_fallback_trigger_rate_by_slice`, and
`max_candidate_fallback_failed_rate`. These checks make fallback dependency
auditable. A candidate that wins mostly by routing pages to the fallback is a
different claim from a local-first engine with selective fallback, even if both
produce strong output metrics. The per-slice trigger limits matter because a
low aggregate fallback rate can still hide misuse on ordinary English or
Nepali text pages.

When an experiment records `environment.fallback_min_quality_score`,
`review-claim` also expects candidate rows to include
`fallback_primary_quality_score` and `fallback_min_quality_score` whenever
`require_candidate_fallback_metrics` is enabled. That keeps quality-gated
fallback policies auditable per sample instead of only recording the threshold
once in `experiment.json`.

Fallback model cards are audited separately from the primary candidate card.
When `experiment.json` records `fallback_model_config` and
`fallback_model_config_sha256`, `review-claim` verifies the hash and runs
`audit-model` on that fallback card. The SOTA profile also sets
`require_fallback_model_present: true`,
`require_candidate_fallback_provenance: true`,
`require_shared_fallback_model: true` and
`require_fallback_model_audit_passed: true`, so a portfolio cannot silently mix
unaudited fallback components across required experiment slots or omit fallback
provenance while still claiming the selective-fallback engine.

The candidate card must also declare the fallback component it was built to use:

```json
{
  "provenance": {
    "fallback_model_card": {
      "path": "runs/fallback/model-card.json",
      "sha256": "..."
    }
  }
}
```

`review-claim` hashes that model card, compares it with every reviewed
`fallback_model_config_sha256`, and audits the fallback card. This closes the
gap where an experiment could record a fallback at benchmark time while the
candidate card itself remained ambiguous about which selective-fallback engine
was being claimed.

That second guardrail matters for exactly the case we hit on `k2`: a rendered-document `script_select_composite` calibrated on `claimpack-rendered-v2` chose `primary_script_threshold=0.0`, meaning the route degenerated to "always pick Surya." Separation from the final holdout is not enough there; the candidate is still not a meaningful new routed model.

The holdout-equivalence guardrail covers the next failure mode after calibration separation: even when calibration was done on a separate split, a routed candidate can still reduce to one component baseline on the reviewed holdout. If the paired benchmark rows show `0` wins, `0` losses, and all ties on CER against a component baseline, `review-claim` now fails `candidate-distinctness`.

For the SOTA track, set `require_trained_recognizer: true`. A direct
`paddleocr` or `hf_vision_encoder_decoder` candidate passes this check only when
the model card has recorded artifacts plus `provenance.train_manifest`,
`provenance.eval_manifest`, and `provenance.training_summary` with SHA-256
hashes. Review resolves those paths from the model-card location and verifies
the files still match the recorded hashes. Composite candidates must point to
the trained component with:

```json
{
  "provenance": {
    "trained_recognizer_model_card": {
      "path": "runs/recognizer/package/model-card.json",
      "sha256": "..."
    }
  }
}
```

This prevents a wrapper around Tesseract, Surya, or a VLM from satisfying the
paper objective by benchmark routing alone.

For recognizer branch admission, set `require_model_admission: true` and record
the admission validation report in the model card:

```json
{
  "metrics": {
    "admission_validation": {
      "path": "outputs/validation-next-recognizer-hard50/validation.json",
      "sha256": "...",
      "required_status": "validated"
    }
  }
}
```

This catches the failure mode we hit with the PaddleOCR branches: the model can
have artifacts and train/eval provenance, but still be empirically rejected by
the hard recognizer gate. Rejected admission reports are summarized with their
failed requirements and gate deltas, so review output can show why a recognizer
was rejected without manually opening the raw validation JSON. Admission reports
are also bound back to `provenance.candidate_model_config`; review rejects a
passing admission artifact if it points at a different model id, backend, base
model, or artifact set.

For document-engine SOTA claims, metric presence is not enough. The
`configs/claim-review-sota-v1.json` portfolio requires passing validation gates
for the named claim dimensions:

- OCR: CER and WER gates on Nepali/English slices where applicable
- tables: `table_cell_f1`
- reading order: `reading_order_pair_accuracy`
- figures: `figure_detection_f1` and `figure_caption_cer`
- reconstruction: `document_reconstruction_similarity`

The corresponding validation configs must contain matching gates. The
`tests/test_claim_review_sota_config.py` checks pin this relationship so a
future config edit cannot silently downgrade figure/caption or WER from a
passing gate to a reported-only metric.

The SOTA profile also requires exported structured-output artifacts from the
candidate benchmark rows:

- every required slot: `document.json`, `document.md`, and `quality.json`
- table slices: table CSV and HTML exports under `tables/`
- figure slices: `figures/metadata.json` and exported figure image files

For current benchmark reports, `review-claim` uses the candidate row's
`output_artifacts` records and verifies each recorded path, SHA-256 hash, and
byte size before inspecting content. Older reports without `output_artifacts`
fall back to path inference from the benchmark report directory and candidate
row input stem. JSON artifacts must parse with the expected top-level shape,
Markdown/table/image files must be nonempty, and figure metadata must contain
entries. This means a claim cannot be supported only by scalar metrics, stale
files, or swapped artifacts if the human-readable Markdown, machine-readable
JSON, tables, or figures were not actually produced as usable artifacts by the
reviewed benchmark row.

## Example Config

```json
{
  "require_trained_recognizer": true,
  "require_model_admission": true,
  "require_fallback_model_present": true,
  "require_candidate_fallback_provenance": true,
  "require_shared_fallback_model": true,
  "require_fallback_model_audit_passed": true,
  "required_experiments": [
    {
      "name": "internal-nepali-pack",
      "datasets": ["ocrtech-eval-pack"],
      "require_validation_status": "validated",
      "require_baselines": ["candidate", "tesseract", "stock-paddle", "glm-ocr", "paddleocr-vl"],
      "required_metrics": ["cer", "wer", "document_reconstruction_similarity", "table_cell_f1", "reading_order_pair_accuracy", "figure_detection_f1", "figure_caption_cer"],
      "required_metric_slices": {
        "cer": ["nepali", "english"],
        "wer": ["nepali", "english"],
        "document_reconstruction_similarity": ["table", "reading_order", "figure"],
        "table_cell_f1": ["table"],
        "reading_order_pair_accuracy": ["reading_order"],
        "figure_detection_f1": ["figure"],
        "figure_caption_cer": ["figure"]
      },
      "required_validation_gates": [
        {"baseline": "tesseract", "metric": "cer", "slices": ["nepali"]},
        {"baseline": "stock-paddle", "metric": "cer", "slices": ["nepali"]},
        {"baseline": "glm-ocr", "metric": "cer", "slices": ["nepali"]},
        {"baseline": "surya", "metric": "table_cell_f1", "slices": ["table"]},
        {"baseline": "paddleocr-vl", "metric": "reading_order_pair_accuracy", "slices": ["reading_order"]}
      ],
      "require_candidate_fallback_metrics": true,
      "max_candidate_fallback_trigger_rate": 0.35,
      "max_candidate_fallback_failed_rate": 0.0,
      "required_slices": ["nepali", "english", "table", "reading_order", "figure", "scan"]
    },
    {
      "name": "ocrbench-v2",
      "datasets": ["ocrbench-v2"],
      "require_validation_status": "validated",
      "required_metrics": ["cer", "wer"],
      "require_baselines": ["candidate", "glm-ocr", "paddleocr-vl"]
    },
    {
      "name": "omnidocbench",
      "datasets": ["omnidocbench"],
      "require_validation_status": "validated",
      "required_metrics": ["cer", "document_reconstruction_similarity", "table_cell_f1", "reading_order_pair_accuracy"],
      "require_baselines": ["candidate", "paddleocr-vl"]
    }
  ]
}
```

The config is intentionally explicit. If two experiments match one slot, the review fails. Tighten the config until every required slot maps to one concrete artifact.
`eval_manifest_patterns` use shell-style wildcards, so a slot such as
`"*Real5-OmniDocBench*"` matches a manifest path containing that benchmark name.

The checked-in SOTA profile is paired with slot-specific validation configs:

- `configs/validation-internal-sota-v1.json`
- `configs/validation-public-omnidocbench-sota-v1.json`
- `configs/validation-public-real5-sota-v1.json`

Those configs are expected to produce the passed gates referenced by
`required_validation_gates`.

## Run

```bash
ocrtech review-claim \
  --experiment outputs/internal-claim/experiment.json \
  --experiment outputs/ocrbench-v2/experiment.json \
  --experiment outputs/omnidocbench/experiment.json \
  --config claim-review.json \
  --model-config runs/recognizer/package/model-card.json \
  --out outputs/claim-review
```

## Outputs

```text
outputs/claim-review/
  claim-review.json
  claim-review.md
```

If this report fails, the claim is not review-ready.

Do not hand-edit copied `experiment.json` files to point at convenient local
paths. If an experiment was run remotely, sync the benchmark report, manifest,
references, model card, and validation report together, or regenerate the
experiment report from the synced paths. `review-claim` intentionally fails on
stale paths instead of guessing because path rewrites without hashes destroy
claim provenance.