|
Download docs/CLAIM_REVIEW.md from prosperious/ocr-tech-k2: direct link, hf CLI and curl.
- Browser
- Download file 16.1 kB
-
https://huggingface.co/prosperious/ocr-tech-k2/resolve/main/docs/CLAIM_REVIEW.md
- Command line
-
hf download hf://prosperious/ocr-tech-k2/docs/CLAIM_REVIEW.md
-
curl -L -o CLAIM_REVIEW.md https://huggingface.co/prosperious/ocr-tech-k2/resolve/main/docs/CLAIM_REVIEW.md
16.1 kB
| # Claim Review | |
| `review-claim` is the portfolio-level gate for a SOTA claim. `validate-claim` can tell you whether one experiment passed its configured gates. It cannot tell you whether you have enough evidence across the internal Nepali/English pack and public suites to say "better than what is out there." That is what `review-claim` is for. | |
| ## What It Audits | |
| `review-claim` checks that: | |
| - every required experiment slot is backed by a real `experiment.json` | |
| - every referenced eval manifest and benchmark report still exists at the | |
| recorded path, and the eval manifest matches the experiment's recorded | |
| SHA-256 | |
| - successful benchmark rows are bound back to sample ids and input paths in the | |
| reviewed eval manifest, and row slice labels must match the manifest entry | |
| - the expected datasets or manifest patterns are present | |
| - required baselines were actually run | |
| - required metric families are present on successful rows for the required baselines | |
| - required metric families are present on the slices they are supposed to | |
| substantiate when `required_metric_slices` is configured | |
| - benchmark reports do not contain error rows when the slot requires clean runs; | |
| status counts are derived from `benchmark/report.json`, not trusted from the | |
| copied `experiment.json` summary | |
| - required validation status is present for internal claim-grade experiments, | |
| and the referenced `validation.json` agrees with `experiment.json` | |
| - required validation gates passed for the baseline/metric/slice comparisons | |
| declared by `required_validation_gates` | |
| - when configured, candidate fallback diagnostics are present and the aggregate | |
| fallback trigger/failure rates stay under the slot limits | |
| - required slices exist for the internal Nepali/English pack | |
| - candidate experiments all point to the same candidate model config | |
| - the candidate model card matches experiment hashes and passes `audit-model`, | |
| with or without an explicit `--model-config` | |
| - if candidate experiments record a fallback model card, that card exists, | |
| matches the experiment SHA-256, and passes `audit-model` | |
| - when configured, the candidate model card itself records the same fallback | |
| component in `provenance.fallback_model_card` | |
| - when configured, the candidate model card proves it includes a trained local | |
| recognizer with artifacts, train/eval manifest hashes, and training-summary | |
| provenance | |
| - when configured, the candidate model card references a passing admission | |
| validation report before it can enter portfolio review | |
| - routed candidates such as `quality_select_composite`, `quality_ranked_ensemble`, and `script_select_composite` were calibrated on a separate eval manifest, not on one of the reviewed experiment manifests | |
| - routed candidates also fail review if their selected calibration trial collapses to one engine on every calibration sample | |
| - routed candidates also fail review if a reviewed benchmark shows they are exactly indistinguishable from one of their component baselines on paired holdout CER | |
| Metric coverage is a hard gate because the target claim is a document-understanding | |
| claim, not a plain OCR claim. A required experiment can now declare | |
| `required_metrics`; each metric must appear as a finite value on at least one | |
| successful row for every required baseline in that slot. For the internal | |
| Nepali/English document pack, the SOTA config requires text OCR metrics | |
| (`cer`, `wer`), document reconstruction | |
| (`document_reconstruction_similarity`), table reconstruction | |
| (`table_cell_f1`), reading order (`reading_order_pair_accuracy`), and figure/caption metrics | |
| (`figure_detection_f1`, `figure_caption_cer`). Public table/document slots | |
| declare their own metric set. | |
| For claim-grade slots, also declare `required_metric_slices`. This prevents a | |
| metric from satisfying the portfolio check on the wrong document type. For | |
| example, `table_cell_f1` must be finite on a `table` or document-parsing row, | |
| and `figure_caption_cer` must be finite on a `figure` row. | |
| Metric presence is not enough. When a required slot includes `candidate` and | |
| one or more external baselines, `review-claim` also requires paired metric | |
| coverage on the same sample ids. If `required_metric_slices` is configured, the | |
| pairing is checked on those slices; otherwise it is checked across all rows. | |
| This prevents a SOTA review from passing because candidate metrics and baseline | |
| metrics both exist somewhere in the report but never compare the same documents. | |
| Sample coverage is also a hard gate. Every `require_baselines` entry must have a | |
| successful benchmark row for every sample in the reviewed eval manifest. This | |
| prevents a report from pointing at a hard eval pack while only running an easy | |
| subset. | |
| When a slot declares `require_validation_status`, `review-claim` reads the | |
| referenced validation report and verifies `claim_status`. For `validated`, it | |
| also requires `passed: true`. The copied status inside `experiment.json` is not | |
| enough on its own. The validation artifact must also carry provenance linking it | |
| to the same benchmark report, eval manifest, validation config, model cards, | |
| and train manifest hashes referenced by the experiment. | |
| Use `required_validation_gates` for claim-grade slots. Metric coverage proves | |
| that a benchmark emitted a metric; passing validation gates prove the candidate | |
| beat a named baseline under the configured sample count, slices, margins, and | |
| statistical checks. Review requires each matched gate to carry quantitative | |
| paired evidence, not just `passed: true`: positive `pairs`, finite candidate | |
| and baseline means, finite absolute improvement, and win/loss/tie counts that | |
| sum to the pair count. It also re-checks the threshold fields embedded in the | |
| gate result, including `min_pairs`, margins, `min_pair_win_rate`, `max_p_value`, | |
| and `require_ci_above_zero`, so a hand-edited report cannot mark a weak gate as | |
| passed without satisfying its own configured test. | |
| The SOTA validation configs also use `min_pair_win_rate` on improvement gates. | |
| This is checked as wins divided by all paired samples, not only non-tied pairs. | |
| It prevents an average-only improvement, or a single win with many ties, from | |
| supporting a broad "better OCR/document engine" claim. | |
| `required_validation_gates` is only valid on slots that also declare | |
| `require_validation_status`. Each required gate baseline must also appear in | |
| `require_baselines`, and each gate metric must appear in `required_metrics`. | |
| For claim-grade slots, set `required_validation_config` to the checked-in | |
| validation config path. `review-claim` hashes that file and requires | |
| `experiment.json` to record the same `validation_config_sha256`. This prevents a | |
| portfolio from passing with an easier local validation config that happens to | |
| contain a subset of the expected gates. | |
| For hybrid candidates, use `require_candidate_fallback_metrics`, | |
| `max_candidate_fallback_trigger_rate`, and | |
| `max_candidate_fallback_failed_rate`. These checks make fallback dependency | |
| auditable. A candidate that wins mostly by routing pages to the fallback is a | |
| different claim from a local-first engine with selective fallback, even if both | |
| produce strong output metrics. | |
| Fallback model cards are audited separately from the primary candidate card. | |
| When `experiment.json` records `fallback_model_config` and | |
| `fallback_model_config_sha256`, `review-claim` verifies the hash and runs | |
| `audit-model` on that fallback card. The SOTA profile also sets | |
| `require_fallback_model_present: true`, | |
| `require_candidate_fallback_provenance: true`, | |
| `require_shared_fallback_model: true` and | |
| `require_fallback_model_audit_passed: true`, so a portfolio cannot silently mix | |
| unaudited fallback components across required experiment slots or omit fallback | |
| provenance while still claiming the selective-fallback engine. | |
| The candidate card must also declare the fallback component it was built to use: | |
| ```json | |
| { | |
| "provenance": { | |
| "fallback_model_card": { | |
| "path": "runs/fallback/model-card.json", | |
| "sha256": "..." | |
| } | |
| } | |
| } | |
| ``` | |
| `review-claim` hashes that model card, compares it with every reviewed | |
| `fallback_model_config_sha256`, and audits the fallback card. This closes the | |
| gap where an experiment could record a fallback at benchmark time while the | |
| candidate card itself remained ambiguous about which selective-fallback engine | |
| was being claimed. | |
| That second guardrail matters for exactly the case we hit on `k2`: a rendered-document `script_select_composite` calibrated on `claimpack-rendered-v2` chose `primary_script_threshold=0.0`, meaning the route degenerated to "always pick Surya." Separation from the final holdout is not enough there; the candidate is still not a meaningful new routed model. | |
| The holdout-equivalence guardrail covers the next failure mode after calibration separation: even when calibration was done on a separate split, a routed candidate can still reduce to one component baseline on the reviewed holdout. If the paired benchmark rows show `0` wins, `0` losses, and all ties on CER against a component baseline, `review-claim` now fails `candidate-distinctness`. | |
| For the SOTA track, set `require_trained_recognizer: true`. A direct | |
| `paddleocr` or `hf_vision_encoder_decoder` candidate passes this check only when | |
| the model card has recorded artifacts plus `provenance.train_manifest`, | |
| `provenance.eval_manifest`, and `provenance.training_summary` with SHA-256 | |
| hashes. Review resolves those paths from the model-card location and verifies | |
| the files still match the recorded hashes. Composite candidates must point to | |
| the trained component with: | |
| ```json | |
| { | |
| "provenance": { | |
| "trained_recognizer_model_card": { | |
| "path": "runs/recognizer/package/model-card.json", | |
| "sha256": "..." | |
| } | |
| } | |
| } | |
| ``` | |
| This prevents a wrapper around Tesseract, Surya, or a VLM from satisfying the | |
| paper objective by benchmark routing alone. | |
| For recognizer branch admission, set `require_model_admission: true` and record | |
| the admission validation report in the model card: | |
| ```json | |
| { | |
| "metrics": { | |
| "admission_validation": { | |
| "path": "outputs/validation-next-recognizer-hard50/validation.json", | |
| "sha256": "...", | |
| "required_status": "validated" | |
| } | |
| } | |
| } | |
| ``` | |
| This catches the failure mode we hit with the PaddleOCR branches: the model can | |
| have artifacts and train/eval provenance, but still be empirically rejected by | |
| the hard recognizer gate. Rejected admission reports are summarized with their | |
| failed requirements and gate deltas, so review output can show why a recognizer | |
| was rejected without manually opening the raw validation JSON. Admission reports | |
| are also bound back to `provenance.candidate_model_config`; review rejects a | |
| passing admission artifact if it points at a different model id, backend, base | |
| model, or artifact set. | |
| For document-engine SOTA claims, metric presence is not enough. The | |
| `configs/claim-review-sota-v1.json` portfolio requires passing validation gates | |
| for the named claim dimensions: | |
| - OCR: CER and WER gates on Nepali/English slices where applicable | |
| - tables: `table_cell_f1` | |
| - reading order: `reading_order_pair_accuracy` | |
| - figures: `figure_detection_f1` and `figure_caption_cer` | |
| - reconstruction: `document_reconstruction_similarity` | |
| The corresponding validation configs must contain matching gates. The | |
| `tests/test_claim_review_sota_config.py` checks pin this relationship so a | |
| future config edit cannot silently downgrade figure/caption or WER from a | |
| passing gate to a reported-only metric. | |
| The SOTA profile also requires exported structured-output artifacts from the | |
| candidate benchmark rows: | |
| - every required slot: `document.json`, `document.md`, and `quality.json` | |
| - table slices: table CSV and HTML exports under `tables/` | |
| - figure slices: `figures/metadata.json` and exported figure image files | |
| `review-claim` infers these paths from the benchmark report directory and the | |
| candidate row input stem. It also performs minimal content checks: JSON | |
| artifacts must parse with the expected top-level shape, Markdown/table/image | |
| files must be nonempty, and figure metadata must contain entries. This means a | |
| claim cannot be supported only by scalar metrics if the human-readable Markdown, | |
| machine-readable JSON, tables, or figures were not actually produced as usable | |
| artifacts. | |
| ## Example Config | |
| ```json | |
| { | |
| "require_trained_recognizer": true, | |
| "require_model_admission": true, | |
| "require_fallback_model_present": true, | |
| "require_candidate_fallback_provenance": true, | |
| "require_shared_fallback_model": true, | |
| "require_fallback_model_audit_passed": true, | |
| "required_experiments": [ | |
| { | |
| "name": "internal-nepali-pack", | |
| "datasets": ["ocrtech-eval-pack"], | |
| "require_validation_status": "validated", | |
| "require_baselines": ["candidate", "tesseract", "stock-paddle", "glm-ocr", "paddleocr-vl"], | |
| "required_metrics": ["cer", "wer", "document_reconstruction_similarity", "table_cell_f1", "reading_order_pair_accuracy", "figure_detection_f1", "figure_caption_cer"], | |
| "required_metric_slices": { | |
| "cer": ["nepali", "english"], | |
| "wer": ["nepali", "english"], | |
| "document_reconstruction_similarity": ["table", "reading_order", "figure"], | |
| "table_cell_f1": ["table"], | |
| "reading_order_pair_accuracy": ["reading_order"], | |
| "figure_detection_f1": ["figure"], | |
| "figure_caption_cer": ["figure"] | |
| }, | |
| "required_validation_gates": [ | |
| {"baseline": "tesseract", "metric": "cer", "slices": ["nepali"]}, | |
| {"baseline": "stock-paddle", "metric": "cer", "slices": ["nepali"]}, | |
| {"baseline": "glm-ocr", "metric": "cer", "slices": ["nepali"]}, | |
| {"baseline": "surya", "metric": "table_cell_f1", "slices": ["table"]}, | |
| {"baseline": "paddleocr-vl", "metric": "reading_order_pair_accuracy", "slices": ["reading_order"]} | |
| ], | |
| "require_candidate_fallback_metrics": true, | |
| "max_candidate_fallback_trigger_rate": 0.35, | |
| "max_candidate_fallback_failed_rate": 0.0, | |
| "required_slices": ["nepali", "english", "table", "reading_order", "figure", "scan"] | |
| }, | |
| { | |
| "name": "ocrbench-v2", | |
| "datasets": ["ocrbench-v2"], | |
| "require_validation_status": "validated", | |
| "required_metrics": ["cer", "wer"], | |
| "require_baselines": ["candidate", "glm-ocr", "paddleocr-vl"] | |
| }, | |
| { | |
| "name": "omnidocbench", | |
| "datasets": ["omnidocbench"], | |
| "require_validation_status": "validated", | |
| "required_metrics": ["cer", "document_reconstruction_similarity", "table_cell_f1", "reading_order_pair_accuracy"], | |
| "require_baselines": ["candidate", "paddleocr-vl"] | |
| } | |
| ] | |
| } | |
| ``` | |
| The config is intentionally explicit. If two experiments match one slot, the review fails. Tighten the config until every required slot maps to one concrete artifact. | |
| `eval_manifest_patterns` use shell-style wildcards, so a slot such as | |
| `"*Real5-OmniDocBench*"` matches a manifest path containing that benchmark name. | |
| The checked-in SOTA profile is paired with slot-specific validation configs: | |
| - `configs/validation-internal-sota-v1.json` | |
| - `configs/validation-public-omnidocbench-sota-v1.json` | |
| - `configs/validation-public-real5-sota-v1.json` | |
| Those configs are expected to produce the passed gates referenced by | |
| `required_validation_gates`. | |
| ## Run | |
| ```bash | |
| ocrtech review-claim \ | |
| --experiment outputs/internal-claim/experiment.json \ | |
| --experiment outputs/ocrbench-v2/experiment.json \ | |
| --experiment outputs/omnidocbench/experiment.json \ | |
| --config claim-review.json \ | |
| --model-config runs/recognizer/package/model-card.json \ | |
| --out outputs/claim-review | |
| ``` | |
| ## Outputs | |
| ```text | |
| outputs/claim-review/ | |
| claim-review.json | |
| claim-review.md | |
| ``` | |
| If this report fails, the claim is not review-ready. | |
| Do not hand-edit copied `experiment.json` files to point at convenient local | |
| paths. If an experiment was run remotely, sync the benchmark report, manifest, | |
| references, model card, and validation report together, or regenerate the | |
| experiment report from the synced paths. `review-claim` intentionally fails on | |
| stale paths instead of guessing because path rewrites without hashes destroy | |
| claim provenance. | |