prosperious's picture
Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified
|
Raw History Blame Contribute Delete
16.9 kB

Claim Review

review-claim is the portfolio-level gate for a SOTA claim. validate-claim can tell you whether one experiment passed its configured gates. It cannot tell you whether you have enough evidence across the internal Nepali/English pack and public suites to say "better than what is out there." That is what review-claim is for.

What It Audits

review-claim checks that:

  • every required experiment slot is backed by a real experiment.json
  • every referenced eval manifest and benchmark report still exists at the recorded path, and the eval manifest matches the experiment's recorded SHA-256
  • successful benchmark rows are bound back to sample ids and input paths in the reviewed eval manifest, and row slice labels must match the manifest entry
  • the expected datasets or manifest patterns are present
  • required baselines were actually run
  • required metric families are present on successful rows for the required baselines
  • required metric families are present on the slices they are supposed to substantiate when required_metric_slices is configured
  • benchmark reports do not contain error rows when the slot requires clean runs; status counts are derived from benchmark/report.json, not trusted from the copied experiment.json summary
  • required validation status is present for internal claim-grade experiments, and the referenced validation.json agrees with experiment.json
  • required validation gates passed for the baseline/metric/slice comparisons declared by required_validation_gates
  • when configured, candidate fallback diagnostics are present and the aggregate fallback trigger/failure rates stay under the slot limits
  • required slices exist for the internal Nepali/English pack
  • candidate experiments all point to the same candidate model config
  • the candidate model card matches experiment hashes and passes audit-model, with or without an explicit --model-config
  • if candidate experiments record a fallback model card, that card exists, matches the experiment SHA-256, and passes audit-model
  • when configured, the candidate model card itself records the same fallback component in provenance.fallback_model_card
  • when configured, the candidate model card proves it includes a trained local recognizer with artifacts, train/eval manifest hashes, and training-summary provenance
  • when configured, the candidate model card references a passing admission validation report before it can enter portfolio review
  • routed candidates such as quality_select_composite, quality_ranked_ensemble, and script_select_composite were calibrated on a separate eval manifest, not on one of the reviewed experiment manifests
  • routed candidates also fail review if their selected calibration trial collapses to one engine on every calibration sample
  • routed candidates also fail review if a reviewed benchmark shows they are exactly indistinguishable from one of their component baselines on paired holdout CER

Metric coverage is a hard gate because the target claim is a document-understanding claim, not a plain OCR claim. A required experiment can now declare required_metrics; each metric must appear as a finite value on at least one successful row for every required baseline in that slot. For the internal Nepali/English document pack, the SOTA config requires text OCR metrics (cer, wer), document reconstruction (document_reconstruction_similarity), table reconstruction (table_cell_f1), reading order (reading_order_pair_accuracy), and figure/caption metrics (figure_detection_f1, figure_caption_cer). Public table/document slots declare their own metric set.

For claim-grade slots, also declare required_metric_slices. This prevents a metric from satisfying the portfolio check on the wrong document type. For example, table_cell_f1 must be finite on a table or document-parsing row, and figure_caption_cer must be finite on a figure row.

Metric presence is not enough. When a required slot includes candidate and one or more external baselines, review-claim also requires paired metric coverage on the same sample ids. If required_metric_slices is configured, the pairing is checked on those slices; otherwise it is checked across all rows. This prevents a SOTA review from passing because candidate metrics and baseline metrics both exist somewhere in the report but never compare the same documents.

Sample coverage is also a hard gate. Every require_baselines entry must have a successful benchmark row for every sample in the reviewed eval manifest. This prevents a report from pointing at a hard eval pack while only running an easy subset.

When a slot declares require_validation_status, review-claim reads the referenced validation report and verifies claim_status. For validated, it also requires passed: true. The copied status inside experiment.json is not enough on its own. The validation artifact must also carry provenance linking it to the same benchmark report, eval manifest, validation config, model cards, and train manifest hashes referenced by the experiment.

Use required_validation_gates for claim-grade slots. Metric coverage proves that a benchmark emitted a metric; passing validation gates prove the candidate beat a named baseline under the configured sample count, slices, margins, and statistical checks. Review requires each matched gate to carry quantitative paired evidence, not just passed: true: positive pairs, finite candidate and baseline means, finite absolute improvement, and win/loss/tie counts that sum to the pair count. It also re-checks the threshold fields embedded in the gate result, including min_pairs, margins, min_pair_win_rate, max_p_value, and require_ci_above_zero, so a hand-edited report cannot mark a weak gate as passed without satisfying its own configured test.

The SOTA validation configs also use min_pair_win_rate on improvement gates. This is checked as wins divided by all paired samples, not only non-tied pairs. It prevents an average-only improvement, or a single win with many ties, from supporting a broad "better OCR/document engine" claim.

required_validation_gates is only valid on slots that also declare require_validation_status. Each required gate baseline must also appear in require_baselines, and each gate metric must appear in required_metrics.

For claim-grade slots, set required_validation_config to the checked-in validation config path. review-claim hashes that file and requires experiment.json to record the same validation_config_sha256. This prevents a portfolio from passing with an easier local validation config that happens to contain a subset of the expected gates.

For hybrid candidates, use require_candidate_fallback_metrics, max_candidate_fallback_trigger_rate, and max_candidate_fallback_trigger_rate_by_slice, and max_candidate_fallback_failed_rate. These checks make fallback dependency auditable. A candidate that wins mostly by routing pages to the fallback is a different claim from a local-first engine with selective fallback, even if both produce strong output metrics. The per-slice trigger limits matter because a low aggregate fallback rate can still hide misuse on ordinary English or Nepali text pages.

When an experiment records environment.fallback_min_quality_score, review-claim also expects candidate rows to include fallback_primary_quality_score and fallback_min_quality_score whenever require_candidate_fallback_metrics is enabled. That keeps quality-gated fallback policies auditable per sample instead of only recording the threshold once in experiment.json.

Fallback model cards are audited separately from the primary candidate card. When experiment.json records fallback_model_config and fallback_model_config_sha256, review-claim verifies the hash and runs audit-model on that fallback card. The SOTA profile also sets require_fallback_model_present: true, require_candidate_fallback_provenance: true, require_shared_fallback_model: true and require_fallback_model_audit_passed: true, so a portfolio cannot silently mix unaudited fallback components across required experiment slots or omit fallback provenance while still claiming the selective-fallback engine.

The candidate card must also declare the fallback component it was built to use:

{
  "provenance": {
    "fallback_model_card": {
      "path": "runs/fallback/model-card.json",
      "sha256": "..."
    }
  }
}

review-claim hashes that model card, compares it with every reviewed fallback_model_config_sha256, and audits the fallback card. This closes the gap where an experiment could record a fallback at benchmark time while the candidate card itself remained ambiguous about which selective-fallback engine was being claimed.

That second guardrail matters for exactly the case we hit on k2: a rendered-document script_select_composite calibrated on claimpack-rendered-v2 chose primary_script_threshold=0.0, meaning the route degenerated to "always pick Surya." Separation from the final holdout is not enough there; the candidate is still not a meaningful new routed model.

The holdout-equivalence guardrail covers the next failure mode after calibration separation: even when calibration was done on a separate split, a routed candidate can still reduce to one component baseline on the reviewed holdout. If the paired benchmark rows show 0 wins, 0 losses, and all ties on CER against a component baseline, review-claim now fails candidate-distinctness.

For the SOTA track, set require_trained_recognizer: true. A direct paddleocr or hf_vision_encoder_decoder candidate passes this check only when the model card has recorded artifacts plus provenance.train_manifest, provenance.eval_manifest, and provenance.training_summary with SHA-256 hashes. Review resolves those paths from the model-card location and verifies the files still match the recorded hashes. Composite candidates must point to the trained component with:

{
  "provenance": {
    "trained_recognizer_model_card": {
      "path": "runs/recognizer/package/model-card.json",
      "sha256": "..."
    }
  }
}

This prevents a wrapper around Tesseract, Surya, or a VLM from satisfying the paper objective by benchmark routing alone.

For recognizer branch admission, set require_model_admission: true and record the admission validation report in the model card:

{
  "metrics": {
    "admission_validation": {
      "path": "outputs/validation-next-recognizer-hard50/validation.json",
      "sha256": "...",
      "required_status": "validated"
    }
  }
}

This catches the failure mode we hit with the PaddleOCR branches: the model can have artifacts and train/eval provenance, but still be empirically rejected by the hard recognizer gate. Rejected admission reports are summarized with their failed requirements and gate deltas, so review output can show why a recognizer was rejected without manually opening the raw validation JSON. Admission reports are also bound back to provenance.candidate_model_config; review rejects a passing admission artifact if it points at a different model id, backend, base model, or artifact set.

For document-engine SOTA claims, metric presence is not enough. The configs/claim-review-sota-v1.json portfolio requires passing validation gates for the named claim dimensions:

  • OCR: CER and WER gates on Nepali/English slices where applicable
  • tables: table_cell_f1
  • reading order: reading_order_pair_accuracy
  • figures: figure_detection_f1 and figure_caption_cer
  • reconstruction: document_reconstruction_similarity

The corresponding validation configs must contain matching gates. The tests/test_claim_review_sota_config.py checks pin this relationship so a future config edit cannot silently downgrade figure/caption or WER from a passing gate to a reported-only metric.

The SOTA profile also requires exported structured-output artifacts from the candidate benchmark rows:

  • every required slot: document.json, document.md, and quality.json
  • table slices: table CSV and HTML exports under tables/
  • figure slices: figures/metadata.json and exported figure image files

For current benchmark reports, review-claim uses the candidate row's output_artifacts records and verifies each recorded path, SHA-256 hash, and byte size before inspecting content. Older reports without output_artifacts fall back to path inference from the benchmark report directory and candidate row input stem. JSON artifacts must parse with the expected top-level shape, Markdown/table/image files must be nonempty, and figure metadata must contain entries. This means a claim cannot be supported only by scalar metrics, stale files, or swapped artifacts if the human-readable Markdown, machine-readable JSON, tables, or figures were not actually produced as usable artifacts by the reviewed benchmark row.

Example Config

{
  "require_trained_recognizer": true,
  "require_model_admission": true,
  "require_fallback_model_present": true,
  "require_candidate_fallback_provenance": true,
  "require_shared_fallback_model": true,
  "require_fallback_model_audit_passed": true,
  "required_experiments": [
    {
      "name": "internal-nepali-pack",
      "datasets": ["ocrtech-eval-pack"],
      "require_validation_status": "validated",
      "require_baselines": ["candidate", "tesseract", "stock-paddle", "glm-ocr", "paddleocr-vl"],
      "required_metrics": ["cer", "wer", "document_reconstruction_similarity", "table_cell_f1", "reading_order_pair_accuracy", "figure_detection_f1", "figure_caption_cer"],
      "required_metric_slices": {
        "cer": ["nepali", "english"],
        "wer": ["nepali", "english"],
        "document_reconstruction_similarity": ["table", "reading_order", "figure"],
        "table_cell_f1": ["table"],
        "reading_order_pair_accuracy": ["reading_order"],
        "figure_detection_f1": ["figure"],
        "figure_caption_cer": ["figure"]
      },
      "required_validation_gates": [
        {"baseline": "tesseract", "metric": "cer", "slices": ["nepali"]},
        {"baseline": "stock-paddle", "metric": "cer", "slices": ["nepali"]},
        {"baseline": "glm-ocr", "metric": "cer", "slices": ["nepali"]},
        {"baseline": "surya", "metric": "table_cell_f1", "slices": ["table"]},
        {"baseline": "paddleocr-vl", "metric": "reading_order_pair_accuracy", "slices": ["reading_order"]}
      ],
      "require_candidate_fallback_metrics": true,
      "max_candidate_fallback_trigger_rate": 0.35,
      "max_candidate_fallback_failed_rate": 0.0,
      "required_slices": ["nepali", "english", "table", "reading_order", "figure", "scan"]
    },
    {
      "name": "ocrbench-v2",
      "datasets": ["ocrbench-v2"],
      "require_validation_status": "validated",
      "required_metrics": ["cer", "wer"],
      "require_baselines": ["candidate", "glm-ocr", "paddleocr-vl"]
    },
    {
      "name": "omnidocbench",
      "datasets": ["omnidocbench"],
      "require_validation_status": "validated",
      "required_metrics": ["cer", "document_reconstruction_similarity", "table_cell_f1", "reading_order_pair_accuracy"],
      "require_baselines": ["candidate", "paddleocr-vl"]
    }
  ]
}

The config is intentionally explicit. If two experiments match one slot, the review fails. Tighten the config until every required slot maps to one concrete artifact. eval_manifest_patterns use shell-style wildcards, so a slot such as "*Real5-OmniDocBench*" matches a manifest path containing that benchmark name.

The checked-in SOTA profile is paired with slot-specific validation configs:

  • configs/validation-internal-sota-v1.json
  • configs/validation-public-omnidocbench-sota-v1.json
  • configs/validation-public-real5-sota-v1.json

Those configs are expected to produce the passed gates referenced by required_validation_gates.

Run

ocrtech review-claim \
  --experiment outputs/internal-claim/experiment.json \
  --experiment outputs/ocrbench-v2/experiment.json \
  --experiment outputs/omnidocbench/experiment.json \
  --config claim-review.json \
  --model-config runs/recognizer/package/model-card.json \
  --out outputs/claim-review

Outputs

outputs/claim-review/
  claim-review.json
  claim-review.md

If this report fails, the claim is not review-ready.

Do not hand-edit copied experiment.json files to point at convenient local paths. If an experiment was run remotely, sync the benchmark report, manifest, references, model card, and validation report together, or regenerate the experiment report from the synced paths. review-claim intentionally fails on stale paths instead of guessing because path rewrites without hashes destroy claim provenance.