|
Download ocr-tech/docs/PREFLIGHT.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 8.44 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/PREFLIGHT.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/PREFLIGHT.md
-
curl -L -o PREFLIGHT.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/PREFLIGHT.md
8.44 kB
| # Claim Preflight | |
| `preflight-claim` is the gate before any expensive benchmark or training-adjacent claim run. Use it to fail on missing references, bad eval packs, missing candidate model cards, missing baseline dependencies, invalid validation configs, train/eval leakage, and tuned-on-eval routed candidates before you burn remote machine time. | |
| ## Claim-Grade Run | |
| ```bash | |
| ocrtech preflight-claim \ | |
| --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ | |
| --references-dir data/eval-pack/references \ | |
| --baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \ | |
| --candidate-model-config runs/recognizer/package/model-card.json \ | |
| --fallback-model-config runs/vlm-fallback/package/model-card.json \ | |
| --require-trained-recognizer \ | |
| --require-model-admission \ | |
| --gorkhapatra-review-audit outputs/gorkhapatra-language-page-review-audit/gorkhapatra-language-page-review-audit.json \ | |
| --validation-config validation.json \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --out outputs/preflight | |
| ``` | |
| This exits with `0` only when every required check passes. | |
| ## Smoke Pack Exception | |
| ```bash | |
| ocrtech preflight-claim \ | |
| --eval-manifest data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \ | |
| --references-dir data/eval-pack-smoke/references \ | |
| --baselines ours \ | |
| --candidate-model-config runs/fixtures/sidecar-model.json \ | |
| --allow-smoke-eval-pack \ | |
| --out outputs/preflight-smoke | |
| ``` | |
| `--allow-smoke-eval-pack` only downgrades the non-image eval-pack check. It does not hide missing references, invalid model cards, missing baseline hooks, or leakage. | |
| ## Outputs | |
| ```text | |
| outputs/preflight/ | |
| preflight.json | |
| preflight.md | |
| eval-pack-audit/eval-pack-audit.json | |
| eval-pack-audit/eval-pack-audit.md | |
| ``` | |
| ## Checks | |
| `preflight-claim` currently verifies: | |
| - eval manifest loads, every input file exists, and recorded input/text hashes | |
| still match the manifest contents | |
| - image/PDF claim inputs use OCR-supported suffixes; text sidecar smoke inputs | |
| are allowed only when `metadata.input_format = text` | |
| - references resolve and parse for every sample | |
| - verified Gorkhapatra/Naya Nepal language-page eval rows have passed manual | |
| disambiguation evidence via `--gorkhapatra-review-audit`; pending | |
| `language_page_candidate` rows fail preflight, and review-audit JSON must have | |
| passed with `--require-verified-references` | |
| - ocrtech eval packs are claim-ready unless `--allow-smoke-eval-pack` is set | |
| - `ours` or `candidate` baselines have a passing candidate model card | |
| - configured fallback model cards pass `audit-model` before the benchmark starts | |
| - when a fallback model is configured, the candidate model card must also | |
| declare the same fallback card in `provenance.fallback_model_card`; preflight | |
| hashes both cards and fails if they differ | |
| - claim runs can require trained-recognizer provenance with | |
| `--require-trained-recognizer`; this checks for recorded trained artifacts and | |
| hash-verified train/eval manifest and training-summary provenance, or a | |
| nested `provenance.trained_recognizer_model_card` in composite candidates | |
| - claim runs can require model admission validation with | |
| `--require-model-admission`; this checks the candidate model card's | |
| `metrics.admission_validation` report and requires `claim_status=validated`. | |
| It also requires quantitative passing gate evidence in that validation JSON. | |
| When the report is rejected, preflight now surfaces the failed requirements | |
| and gate deltas from the validation JSON, including baseline, metric, slice, | |
| paired means, and win/loss/tie counts. The admission report must also include | |
| `provenance.candidate_model_config`; preflight compares the validated model | |
| identity and artifact hashes against the current candidate model card so a | |
| passing report from a different recognizer cannot be reused. | |
| - routed candidates such as `quality_select_composite`, `quality_ranked_ensemble`, and `script_select_composite` have calibration provenance, and that calibration eval manifest is not the same as the current claim eval manifest | |
| - routed candidates also fail if their selected calibration trial collapses to one engine for every calibration sample, because that is not a meaningful routed candidate for claim purposes | |
| - routed candidates also fail unless the claim-grade benchmark includes their raw component baselines, so distinctness can be measured in the reviewed evidence | |
| - `tesseract` is on `PATH` when requested | |
| - `paddleocr` is importable when stock PaddleOCR is requested | |
| - `OCRTECH_GLM_OCR_CMD` is set, shell-parseable, contains both `{input}` and `{out}`, and points at a runnable executable when `glm-ocr` is requested | |
| - `OCRTECH_PADDLEOCR_VL_CMD` is set, shell-parseable, contains both `{input}` and `{out}`, and points at a runnable executable when `paddleocr-vl` is requested | |
| - validation config loads, contains gates, names baselines that are scheduled | |
| for the run, and can be satisfied by the eval manifest's sample and slice | |
| counts | |
| - train/eval leakage passes when train manifests are supplied | |
| For a reusable reference-only artifact, run: | |
| ```bash | |
| ocrtech audit-references \ | |
| data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ | |
| --out outputs/reference-audit | |
| ``` | |
| For manually verified Gorkhapatra/Naya Nepal references, add | |
| `--require-claim-ready`. `preflight-claim` now uses the same reference parser as | |
| benchmark scoring, so malformed structured references fail before engines run. | |
| Use this before `run-experiment`. It is cheaper to fail here than after a half-day benchmark on a broken setup. | |
| For a single guarded command, `run-experiment` can run the same preflight before | |
| benchmarking: | |
| ```bash | |
| ocrtech run-experiment \ | |
| --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ | |
| --references-dir data/eval-pack/references \ | |
| --baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \ | |
| --candidate-model-config runs/recognizer/package/model-card.json \ | |
| --validation-config validation.json \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --preflight \ | |
| --require-trained-recognizer \ | |
| --require-model-admission \ | |
| --out outputs/claim-experiment \ | |
| --require-validated | |
| ``` | |
| When embedded preflight fails, `run-experiment` writes | |
| `outputs/claim-experiment/preflight/preflight.json` and stops before creating a | |
| benchmark report. | |
| When `--validation-config` is omitted, preflight emits a warning instead of | |
| claiming the default gates are appropriate for the run. For claim-grade work, | |
| always pass the slot-specific validation config. When a config is supplied, | |
| preflight rejects impossible setups before benchmark time: `min_samples` larger | |
| than the manifest, required slices absent from the manifest, gate `min_pairs` | |
| larger than the matching slice intersection, or gate baselines that are not in | |
| `--baselines`. | |
| If the `eval-input-suffixes` check fails on files such as `.img`, normalize the | |
| manifest before rerunning preflight: | |
| ```bash | |
| ocrtech normalize-manifest-images \ | |
| data/splits-mixed-v1/eval-with-refs.jsonl \ | |
| --out data/splits-mixed-v1/eval-normalized-images | |
| ocrtech materialize-references \ | |
| data/splits-mixed-v1/eval-normalized-images/manifest-images-normalized.jsonl \ | |
| --out data/references-mixed-v1-normalized \ | |
| --rewritten-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl | |
| ``` | |
| For routed thresholded candidates, the intended behavior is now: | |
| - calibrate on one experiment manifest with `ocrtech calibrate-quality-router` | |
| - or `ocrtech calibrate-script-router` | |
| - benchmark with `candidate` plus the candidate's raw component baselines in the same run | |
| - benchmark on a different eval manifest | |
| - expect `candidate-calibration=pass` only on that different eval manifest | |
| On `k2`, the same-pack check was verified against a real routed Tesseract calibration artifact: preflight failed on the calibration manifest itself and passed the calibration-separation check on the untouched holdout manifest. | |
| On the larger rendered `claimpack-rendered-v2` holdout, a second real case now exists: the calibrated script router selected `primary_script_threshold=0.0`, which meant `secondary_picks=50/50` on calibration. After syncing the new guardrail to `k2`, `preflight-claim` correctly failed that model with: | |
| - `script_select_composite collapses to the secondary engine on every calibration sample` | |
| That is the intended behavior. A routed candidate that reduces to "always use Surya" is not evidence of a novel better engine. | |