Download ocr-tech-mps-integration/docs/PREFLIGHT.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 8.44 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/PREFLIGHT.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech-mps-integration/docs/PREFLIGHT.md
-
curl -L -o PREFLIGHT.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/PREFLIGHT.md
Claim Preflight
preflight-claim is the gate before any expensive benchmark or training-adjacent claim run. Use it to fail on missing references, bad eval packs, missing candidate model cards, missing baseline dependencies, invalid validation configs, train/eval leakage, and tuned-on-eval routed candidates before you burn remote machine time.
Claim-Grade Run
ocrtech preflight-claim \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--candidate-model-config runs/recognizer/package/model-card.json \
--fallback-model-config runs/vlm-fallback/package/model-card.json \
--require-trained-recognizer \
--require-model-admission \
--gorkhapatra-review-audit outputs/gorkhapatra-language-page-review-audit/gorkhapatra-language-page-review-audit.json \
--validation-config validation.json \
--train-manifest data/splits/train.jsonl \
--out outputs/preflight
This exits with 0 only when every required check passes.
Smoke Pack Exception
ocrtech preflight-claim \
--eval-manifest data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack-smoke/references \
--baselines ours \
--candidate-model-config runs/fixtures/sidecar-model.json \
--allow-smoke-eval-pack \
--out outputs/preflight-smoke
--allow-smoke-eval-pack only downgrades the non-image eval-pack check. It does not hide missing references, invalid model cards, missing baseline hooks, or leakage.
Outputs
outputs/preflight/
preflight.json
preflight.md
eval-pack-audit/eval-pack-audit.json
eval-pack-audit/eval-pack-audit.md
Checks
preflight-claim currently verifies:
- eval manifest loads, every input file exists, and recorded input/text hashes still match the manifest contents
- image/PDF claim inputs use OCR-supported suffixes; text sidecar smoke inputs
are allowed only when
metadata.input_format = text - references resolve and parse for every sample
- verified Gorkhapatra/Naya Nepal language-page eval rows have passed manual
disambiguation evidence via
--gorkhapatra-review-audit; pendinglanguage_page_candidaterows fail preflight, and review-audit JSON must have passed with--require-verified-references - ocrtech eval packs are claim-ready unless
--allow-smoke-eval-packis set oursorcandidatebaselines have a passing candidate model card- configured fallback model cards pass
audit-modelbefore the benchmark starts - when a fallback model is configured, the candidate model card must also
declare the same fallback card in
provenance.fallback_model_card; preflight hashes both cards and fails if they differ - claim runs can require trained-recognizer provenance with
--require-trained-recognizer; this checks for recorded trained artifacts and hash-verified train/eval manifest and training-summary provenance, or a nestedprovenance.trained_recognizer_model_cardin composite candidates - claim runs can require model admission validation with
--require-model-admission; this checks the candidate model card'smetrics.admission_validationreport and requiresclaim_status=validated. It also requires quantitative passing gate evidence in that validation JSON. When the report is rejected, preflight now surfaces the failed requirements and gate deltas from the validation JSON, including baseline, metric, slice, paired means, and win/loss/tie counts. The admission report must also includeprovenance.candidate_model_config; preflight compares the validated model identity and artifact hashes against the current candidate model card so a passing report from a different recognizer cannot be reused. - routed candidates such as
quality_select_composite,quality_ranked_ensemble, andscript_select_compositehave calibration provenance, and that calibration eval manifest is not the same as the current claim eval manifest - routed candidates also fail if their selected calibration trial collapses to one engine for every calibration sample, because that is not a meaningful routed candidate for claim purposes
- routed candidates also fail unless the claim-grade benchmark includes their raw component baselines, so distinctness can be measured in the reviewed evidence
tesseractis onPATHwhen requestedpaddleocris importable when stock PaddleOCR is requestedOCRTECH_GLM_OCR_CMDis set, shell-parseable, contains both{input}and{out}, and points at a runnable executable whenglm-ocris requestedOCRTECH_PADDLEOCR_VL_CMDis set, shell-parseable, contains both{input}and{out}, and points at a runnable executable whenpaddleocr-vlis requested- validation config loads, contains gates, names baselines that are scheduled for the run, and can be satisfied by the eval manifest's sample and slice counts
- train/eval leakage passes when train manifests are supplied
For a reusable reference-only artifact, run:
ocrtech audit-references \
data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--out outputs/reference-audit
For manually verified Gorkhapatra/Naya Nepal references, add
--require-claim-ready. preflight-claim now uses the same reference parser as
benchmark scoring, so malformed structured references fail before engines run.
Use this before run-experiment. It is cheaper to fail here than after a half-day benchmark on a broken setup.
For a single guarded command, run-experiment can run the same preflight before
benchmarking:
ocrtech run-experiment \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--candidate-model-config runs/recognizer/package/model-card.json \
--validation-config validation.json \
--train-manifest data/splits/train.jsonl \
--preflight \
--require-trained-recognizer \
--require-model-admission \
--out outputs/claim-experiment \
--require-validated
When embedded preflight fails, run-experiment writes
outputs/claim-experiment/preflight/preflight.json and stops before creating a
benchmark report.
When --validation-config is omitted, preflight emits a warning instead of
claiming the default gates are appropriate for the run. For claim-grade work,
always pass the slot-specific validation config. When a config is supplied,
preflight rejects impossible setups before benchmark time: min_samples larger
than the manifest, required slices absent from the manifest, gate min_pairs
larger than the matching slice intersection, or gate baselines that are not in
--baselines.
If the eval-input-suffixes check fails on files such as .img, normalize the
manifest before rerunning preflight:
ocrtech normalize-manifest-images \
data/splits-mixed-v1/eval-with-refs.jsonl \
--out data/splits-mixed-v1/eval-normalized-images
ocrtech materialize-references \
data/splits-mixed-v1/eval-normalized-images/manifest-images-normalized.jsonl \
--out data/references-mixed-v1-normalized \
--rewritten-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl
For routed thresholded candidates, the intended behavior is now:
- calibrate on one experiment manifest with
ocrtech calibrate-quality-router - or
ocrtech calibrate-script-router - benchmark with
candidateplus the candidate's raw component baselines in the same run - benchmark on a different eval manifest
- expect
candidate-calibration=passonly on that different eval manifest
On k2, the same-pack check was verified against a real routed Tesseract calibration artifact: preflight failed on the calibration manifest itself and passed the calibration-separation check on the untouched holdout manifest.
On the larger rendered claimpack-rendered-v2 holdout, a second real case now exists: the calibrated script router selected primary_script_threshold=0.0, which meant secondary_picks=50/50 on calibration. After syncing the new guardrail to k2, preflight-claim correctly failed that model with:
script_select_composite collapses to the secondary engine on every calibration sample
That is the intended behavior. A routed candidate that reduces to "always use Surya" is not evidence of a novel better engine.