prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
8.44 kB

Claim Preflight

preflight-claim is the gate before any expensive benchmark or training-adjacent claim run. Use it to fail on missing references, bad eval packs, missing candidate model cards, missing baseline dependencies, invalid validation configs, train/eval leakage, and tuned-on-eval routed candidates before you burn remote machine time.

Claim-Grade Run

ocrtech preflight-claim \
  --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --references-dir data/eval-pack/references \
  --baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
  --candidate-model-config runs/recognizer/package/model-card.json \
  --fallback-model-config runs/vlm-fallback/package/model-card.json \
  --require-trained-recognizer \
  --require-model-admission \
  --gorkhapatra-review-audit outputs/gorkhapatra-language-page-review-audit/gorkhapatra-language-page-review-audit.json \
  --validation-config validation.json \
  --train-manifest data/splits/train.jsonl \
  --out outputs/preflight

This exits with 0 only when every required check passes.

Smoke Pack Exception

ocrtech preflight-claim \
  --eval-manifest data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
  --references-dir data/eval-pack-smoke/references \
  --baselines ours \
  --candidate-model-config runs/fixtures/sidecar-model.json \
  --allow-smoke-eval-pack \
  --out outputs/preflight-smoke

--allow-smoke-eval-pack only downgrades the non-image eval-pack check. It does not hide missing references, invalid model cards, missing baseline hooks, or leakage.

Outputs

outputs/preflight/
  preflight.json
  preflight.md
  eval-pack-audit/eval-pack-audit.json
  eval-pack-audit/eval-pack-audit.md

Checks

preflight-claim currently verifies:

  • eval manifest loads, every input file exists, and recorded input/text hashes still match the manifest contents
  • image/PDF claim inputs use OCR-supported suffixes; text sidecar smoke inputs are allowed only when metadata.input_format = text
  • references resolve and parse for every sample
  • verified Gorkhapatra/Naya Nepal language-page eval rows have passed manual disambiguation evidence via --gorkhapatra-review-audit; pending language_page_candidate rows fail preflight, and review-audit JSON must have passed with --require-verified-references
  • ocrtech eval packs are claim-ready unless --allow-smoke-eval-pack is set
  • ours or candidate baselines have a passing candidate model card
  • configured fallback model cards pass audit-model before the benchmark starts
  • when a fallback model is configured, the candidate model card must also declare the same fallback card in provenance.fallback_model_card; preflight hashes both cards and fails if they differ
  • claim runs can require trained-recognizer provenance with --require-trained-recognizer; this checks for recorded trained artifacts and hash-verified train/eval manifest and training-summary provenance, or a nested provenance.trained_recognizer_model_card in composite candidates
  • claim runs can require model admission validation with --require-model-admission; this checks the candidate model card's metrics.admission_validation report and requires claim_status=validated. It also requires quantitative passing gate evidence in that validation JSON. When the report is rejected, preflight now surfaces the failed requirements and gate deltas from the validation JSON, including baseline, metric, slice, paired means, and win/loss/tie counts. The admission report must also include provenance.candidate_model_config; preflight compares the validated model identity and artifact hashes against the current candidate model card so a passing report from a different recognizer cannot be reused.
  • routed candidates such as quality_select_composite, quality_ranked_ensemble, and script_select_composite have calibration provenance, and that calibration eval manifest is not the same as the current claim eval manifest
  • routed candidates also fail if their selected calibration trial collapses to one engine for every calibration sample, because that is not a meaningful routed candidate for claim purposes
  • routed candidates also fail unless the claim-grade benchmark includes their raw component baselines, so distinctness can be measured in the reviewed evidence
  • tesseract is on PATH when requested
  • paddleocr is importable when stock PaddleOCR is requested
  • OCRTECH_GLM_OCR_CMD is set, shell-parseable, contains both {input} and {out}, and points at a runnable executable when glm-ocr is requested
  • OCRTECH_PADDLEOCR_VL_CMD is set, shell-parseable, contains both {input} and {out}, and points at a runnable executable when paddleocr-vl is requested
  • validation config loads, contains gates, names baselines that are scheduled for the run, and can be satisfied by the eval manifest's sample and slice counts
  • train/eval leakage passes when train manifests are supplied

For a reusable reference-only artifact, run:

ocrtech audit-references \
  data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --out outputs/reference-audit

For manually verified Gorkhapatra/Naya Nepal references, add --require-claim-ready. preflight-claim now uses the same reference parser as benchmark scoring, so malformed structured references fail before engines run.

Use this before run-experiment. It is cheaper to fail here than after a half-day benchmark on a broken setup.

For a single guarded command, run-experiment can run the same preflight before benchmarking:

ocrtech run-experiment \
  --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --references-dir data/eval-pack/references \
  --baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
  --candidate-model-config runs/recognizer/package/model-card.json \
  --validation-config validation.json \
  --train-manifest data/splits/train.jsonl \
  --preflight \
  --require-trained-recognizer \
  --require-model-admission \
  --out outputs/claim-experiment \
  --require-validated

When embedded preflight fails, run-experiment writes outputs/claim-experiment/preflight/preflight.json and stops before creating a benchmark report.

When --validation-config is omitted, preflight emits a warning instead of claiming the default gates are appropriate for the run. For claim-grade work, always pass the slot-specific validation config. When a config is supplied, preflight rejects impossible setups before benchmark time: min_samples larger than the manifest, required slices absent from the manifest, gate min_pairs larger than the matching slice intersection, or gate baselines that are not in --baselines.

If the eval-input-suffixes check fails on files such as .img, normalize the manifest before rerunning preflight:

ocrtech normalize-manifest-images \
  data/splits-mixed-v1/eval-with-refs.jsonl \
  --out data/splits-mixed-v1/eval-normalized-images

ocrtech materialize-references \
  data/splits-mixed-v1/eval-normalized-images/manifest-images-normalized.jsonl \
  --out data/references-mixed-v1-normalized \
  --rewritten-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl

For routed thresholded candidates, the intended behavior is now:

  • calibrate on one experiment manifest with ocrtech calibrate-quality-router
  • or ocrtech calibrate-script-router
  • benchmark with candidate plus the candidate's raw component baselines in the same run
  • benchmark on a different eval manifest
  • expect candidate-calibration=pass only on that different eval manifest

On k2, the same-pack check was verified against a real routed Tesseract calibration artifact: preflight failed on the calibration manifest itself and passed the calibration-separation check on the untouched holdout manifest.

On the larger rendered claimpack-rendered-v2 holdout, a second real case now exists: the calibrated script router selected primary_script_threshold=0.0, which meant secondary_picks=50/50 on calibration. After syncing the new guardrail to k2, preflight-claim correctly failed that model with:

  • script_select_composite collapses to the secondary engine on every calibration sample

That is the intended behavior. A routed candidate that reduces to "always use Surya" is not evidence of a novel better engine.