ocr-tech-k2 / docs /PREFLIGHT.md
prosperious's picture
Add 51 files (.gitignore ...)
1c43956 verified
|
Raw History Blame Contribute Delete
7.59 kB
# Claim Preflight
`preflight-claim` is the gate before any expensive benchmark or training-adjacent claim run. Use it to fail on missing references, bad eval packs, missing candidate model cards, missing baseline dependencies, invalid validation configs, train/eval leakage, and tuned-on-eval routed candidates before you burn remote machine time.
## Claim-Grade Run
```bash
ocrtech preflight-claim \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--candidate-model-config runs/recognizer/package/model-card.json \
--fallback-model-config runs/vlm-fallback/package/model-card.json \
--require-trained-recognizer \
--require-model-admission \
--validation-config validation.json \
--train-manifest data/splits/train.jsonl \
--out outputs/preflight
```
This exits with `0` only when every required check passes.
## Smoke Pack Exception
```bash
ocrtech preflight-claim \
--eval-manifest data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack-smoke/references \
--baselines ours \
--candidate-model-config runs/fixtures/sidecar-model.json \
--allow-smoke-eval-pack \
--out outputs/preflight-smoke
```
`--allow-smoke-eval-pack` only downgrades the non-image eval-pack check. It does not hide missing references, invalid model cards, missing baseline hooks, or leakage.
## Outputs
```text
outputs/preflight/
preflight.json
preflight.md
eval-pack-audit/eval-pack-audit.json
eval-pack-audit/eval-pack-audit.md
```
## Checks
`preflight-claim` currently verifies:
- eval manifest loads and every input file exists
- image/PDF claim inputs use OCR-supported suffixes; text sidecar smoke inputs
are allowed only when `metadata.input_format = text`
- references resolve and parse for every sample
- ocrtech eval packs are claim-ready unless `--allow-smoke-eval-pack` is set
- `ours` or `candidate` baselines have a passing candidate model card
- configured fallback model cards pass `audit-model` before the benchmark starts
- when a fallback model is configured, the candidate model card must also
declare the same fallback card in `provenance.fallback_model_card`; preflight
hashes both cards and fails if they differ
- claim runs can require trained-recognizer provenance with
`--require-trained-recognizer`; this checks for recorded trained artifacts and
hash-verified train/eval manifest and training-summary provenance, or a
nested `provenance.trained_recognizer_model_card` in composite candidates
- claim runs can require model admission validation with
`--require-model-admission`; this checks the candidate model card's
`metrics.admission_validation` report and requires `claim_status=validated`.
It also requires quantitative passing gate evidence in that validation JSON.
When the report is rejected, preflight now surfaces the failed requirements
and gate deltas from the validation JSON, including baseline, metric, slice,
paired means, and win/loss/tie counts. The admission report must also include
`provenance.candidate_model_config`; preflight compares the validated model
identity and artifact hashes against the current candidate model card so a
passing report from a different recognizer cannot be reused.
- routed candidates such as `quality_select_composite`, `quality_ranked_ensemble`, and `script_select_composite` have calibration provenance, and that calibration eval manifest is not the same as the current claim eval manifest
- routed candidates also fail if their selected calibration trial collapses to one engine for every calibration sample, because that is not a meaningful routed candidate for claim purposes
- routed candidates also fail unless the claim-grade benchmark includes their raw component baselines, so distinctness can be measured in the reviewed evidence
- `tesseract` is on `PATH` when requested
- `paddleocr` is importable when stock PaddleOCR is requested
- `OCRTECH_GLM_OCR_CMD` is set, shell-parseable, contains both `{input}` and `{out}`, and points at a runnable executable when `glm-ocr` is requested
- `OCRTECH_PADDLEOCR_VL_CMD` is set, shell-parseable, contains both `{input}` and `{out}`, and points at a runnable executable when `paddleocr-vl` is requested
- validation config loads, contains gates, names baselines that are scheduled
for the run, and can be satisfied by the eval manifest's sample and slice
counts
- train/eval leakage passes when train manifests are supplied
Use this before `run-experiment`. It is cheaper to fail here than after a half-day benchmark on a broken setup.
For a single guarded command, `run-experiment` can run the same preflight before
benchmarking:
```bash
ocrtech run-experiment \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--candidate-model-config runs/recognizer/package/model-card.json \
--validation-config validation.json \
--train-manifest data/splits/train.jsonl \
--preflight \
--require-trained-recognizer \
--require-model-admission \
--out outputs/claim-experiment \
--require-validated
```
When embedded preflight fails, `run-experiment` writes
`outputs/claim-experiment/preflight/preflight.json` and stops before creating a
benchmark report.
When `--validation-config` is omitted, preflight emits a warning instead of
claiming the default gates are appropriate for the run. For claim-grade work,
always pass the slot-specific validation config. When a config is supplied,
preflight rejects impossible setups before benchmark time: `min_samples` larger
than the manifest, required slices absent from the manifest, gate `min_pairs`
larger than the matching slice intersection, or gate baselines that are not in
`--baselines`.
If the `eval-input-suffixes` check fails on files such as `.img`, normalize the
manifest before rerunning preflight:
```bash
ocrtech normalize-manifest-images \
data/splits-mixed-v1/eval-with-refs.jsonl \
--out data/splits-mixed-v1/eval-normalized-images
ocrtech materialize-references \
data/splits-mixed-v1/eval-normalized-images/manifest-images-normalized.jsonl \
--out data/references-mixed-v1-normalized \
--rewritten-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl
```
For routed thresholded candidates, the intended behavior is now:
- calibrate on one experiment manifest with `ocrtech calibrate-quality-router`
- or `ocrtech calibrate-script-router`
- benchmark with `candidate` plus the candidate's raw component baselines in the same run
- benchmark on a different eval manifest
- expect `candidate-calibration=pass` only on that different eval manifest
On `k2`, the same-pack check was verified against a real routed Tesseract calibration artifact: preflight failed on the calibration manifest itself and passed the calibration-separation check on the untouched holdout manifest.
On the larger rendered `claimpack-rendered-v2` holdout, a second real case now exists: the calibrated script router selected `primary_script_threshold=0.0`, which meant `secondary_picks=50/50` on calibration. After syncing the new guardrail to `k2`, `preflight-claim` correctly failed that model with:
- `script_select_composite collapses to the secondary engine on every calibration sample`
That is the intended behavior. A routed candidate that reduces to "always use Surya" is not evidence of a novel better engine.