Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified |
Download ocr-tech-mps-integration/docs/MODELS.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 26.7 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/MODELS.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech-mps-integration/docs/MODELS.md
-
curl -L -o MODELS.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/MODELS.md
26.7 kB
| # Candidate Model Cards | |
| The candidate OCR engine must run from an explicit model card. Do not benchmark `ours` against external systems unless the candidate model card is attached to the experiment. | |
| Supported candidate backends: | |
| - `paddleocr` | |
| - `hf_vision_encoder_decoder` | |
| - `sidecar` | |
| - `tesseract` | |
| - `text_correction_composite` | |
| - `quality_select_composite` | |
| - `quality_ranked_ensemble` | |
| - `script_select_composite` | |
| - `line_align_composite` | |
| - `table_cell_refine_composite` | |
| ## Export And Package | |
| After `train-recognizer` finishes and a PaddleOCR checkpoint exists, export an inference model: | |
| ```bash | |
| ocrtech export-recognizer \ | |
| --training-config runs/recognizer/paddle_recognizer.yml \ | |
| --checkpoint runs/recognizer/checkpoints/best_accuracy \ | |
| --out runs/recognizer/export | |
| ``` | |
| Run the generated export command from a PaddleOCR checkout, or pass `--run` when the checkout exists locally. | |
| Then package the exported inference directory into an auditable candidate model: | |
| ```bash | |
| ocrtech package-recognizer \ | |
| --inference-dir runs/recognizer/export/inference \ | |
| --dictionary runs/recognizer/nep_eng_dict.txt \ | |
| --out runs/recognizer/package \ | |
| --model-id ocrtech-nep-eng-rec-v1 \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --eval-manifest data/splits/eval.jsonl \ | |
| --training-summary runs/recognizer/recognizer-training-summary.json \ | |
| --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json | |
| ``` | |
| This writes `runs/recognizer/package/model-card.json` and copies all required inference artifacts under `runs/recognizer/package/artifacts/`. | |
| Artifact entries record path, kind, SHA-256 for files, and `size_bytes`. | |
| Directory artifacts use the recursive total size. `audit-model` recomputes those | |
| sizes and reports `total_artifact_size_bytes`, which is the model-size value to | |
| carry into benchmark and paper tables. | |
| ## Training Recipe Evidence | |
| `train-recognizer` writes a machine-readable | |
| `recognizer-training-summary.json` beside the backend training config: | |
| `paddle_recognizer.yml` for PaddleOCR and `hf_recognizer_config.json` for | |
| Hugging Face vision-encoder-decoder runs. Preserve it with the recipe, bundle, | |
| training logs, exported inference model, and packaged model card. It records: | |
| - train/eval manifest paths and SHA-256 hashes | |
| - train/eval sample counts, dataset counts, and slice counts | |
| - generated label, dictionary, and config hashes | |
| - dictionary character coverage, including ASCII and Devanagari presence | |
| - the recipe hyperparameters used for launch | |
| When a custom `--dictionary` is supplied, recipe generation now checks it | |
| against the exact normalized PaddleOCR label text for every train and eval | |
| sample. Missing characters fail before a remote training job is packaged. This | |
| prevents a recognizer run from silently launching with labels that the | |
| recognition head cannot emit. | |
| This summary is not a quality result. It is launch provenance for the local | |
| recognizer branch, and it helps distinguish a reproducible recognizer attempt | |
| from an undocumented remote training run. | |
| `bundle-recognizer-training` includes this summary in the archive and records | |
| its path and SHA-256 under `training_summary` in | |
| `recognizer-training-bundle.json`. Check that field after syncing the bundle to | |
| a CUDA host. | |
| For real GPU handoffs, pass the latest recognizer failure review to both recipe | |
| generation and bundling. This rejects families already stopped by admission | |
| evidence before another remote job is launched: | |
| ```bash | |
| ocrtech train-recognizer \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --eval-manifest data/splits/eval.jsonl \ | |
| --out runs/recognizer \ | |
| --failure-review outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json | |
| ocrtech bundle-recognizer-training \ | |
| --recipe-dir runs/recognizer \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --eval-manifest data/splits/eval.jsonl \ | |
| --out runs/recognizer-training-bundle \ | |
| --require-corpus-audit \ | |
| --failure-review outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json | |
| ``` | |
| If that review contains `stop_family` for the recipe backend/base model, the | |
| command fails with the rejected-run and win/loss/tie evidence. Changing the run | |
| name, learning rate, or corpus is not enough to bypass a stopped family; use a | |
| materially different base recognizer, objective, or document-native model. | |
| The Limbu/Sirijonga Namdhinggo augmentation ablation is recorded separately in | |
| `configs/limbu-sirijonga-augmentation-ablation-v1.json`. Treat it as a narrow | |
| script-adaptation experiment, not as a revival of the stopped mixed | |
| Devanagari PP-OCRv3 engine branch. The three recipe directories are: | |
| ```text | |
| runs/sirijonga-limbu-paddle-ablation-clean-v1 | |
| runs/sirijonga-limbu-paddle-ablation-lighting-v1 | |
| runs/sirijonga-limbu-paddle-ablation-camera-v1 | |
| ``` | |
| Those recipes are handoff-ready only after regenerating the ignored manifests | |
| under `data/ablation-limbu-namdhinggo-augmented-v1/` and rerunning the | |
| corresponding `audit-rendered-degradation-split` checks. A trained checkpoint | |
| from this ablation cannot support paper accuracy claims until it is evaluated | |
| on manually reviewed real Gorkhapatra/Naya Nepal Limbu references and checked | |
| against broader English/Nepali/document-structure regressions. | |
| Pass the same file to `package-recognizer --training-summary` or | |
| `package-hf-recognizer --training-summary` so the final `model-card.json` | |
| records the summary path and SHA-256 under `provenance.training_summary`. | |
| ## Hugging Face Packaging | |
| For an arm64-compatible recognizer path, train a Hugging Face vision-text model and package the saved model directory: | |
| ```bash | |
| ocrtech train-recognizer \ | |
| --backend hf-vision-encoder-decoder \ | |
| --base-model microsoft/trocr-small-printed \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --eval-manifest data/splits/eval.jsonl \ | |
| --out runs/hf-recognizer | |
| ocrtech package-hf-recognizer \ | |
| --model-dir runs/hf-recognizer/final-model \ | |
| --out runs/hf-recognizer/package \ | |
| --model-id ocrtech-hf-nep-eng-rec-v1 \ | |
| --base-model microsoft/trocr-small-printed \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --eval-manifest data/splits/eval.jsonl \ | |
| --training-summary runs/hf-recognizer/recognizer-training-summary.json \ | |
| --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json | |
| ``` | |
| The HF recipe writes `runs/hf-recognizer/recognizer-training-summary.json` at | |
| recipe-generation time. That summary records train/eval manifest hashes, | |
| script coverage, config hash, and hyperparameters. Packaging should include it | |
| just like the PaddleOCR path; otherwise the trained-recognizer provenance gate | |
| will have weaker launch evidence. | |
| This writes `runs/hf-recognizer/package/model-card.json` and records backend kwargs for loading the saved processor and model directory. | |
| ## Create Manually | |
| ```bash | |
| ocrtech create-model-card \ | |
| --out runs/recognizer/model-card.json \ | |
| --model-id ocrtech-nep-eng-rec-v1 \ | |
| --backend paddleocr \ | |
| --base-model PaddlePaddle/devanagari_PP-OCRv3_mobile_rec \ | |
| --artifact runs/recognizer/checkpoints/best_accuracy.pdparams \ | |
| --artifact runs/recognizer/nep_eng_dict.txt \ | |
| --paddleocr-kwarg lang=devanagari \ | |
| --paddleocr-kwarg rec_model_dir=runs/recognizer/inference \ | |
| --paddleocr-kwarg rec_char_dict_path=runs/recognizer/nep_eng_dict.txt \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --eval-manifest data/splits/eval.jsonl \ | |
| --metrics-report outputs/claim-experiment/validation/validation.json | |
| ``` | |
| The backend kwargs are passed through to the selected runtime backend. For `paddleocr`, keep them explicit in the model card because PaddleOCR constructor names can change by version. | |
| ## Routed Candidates | |
| For routed or thresholded candidates, do not hand-edit the threshold into a model card and call that evidence. Calibrate it from a separate experiment artifact: | |
| ```bash | |
| ocrtech calibrate-quality-router \ | |
| --experiment-report outputs/tesseract-psm13-calibration/experiment.json \ | |
| --out outputs/quality-router-calibration \ | |
| --model-id ocrtech-tesseract-quality-router \ | |
| --primary-engine tesseract \ | |
| --secondary-engine tesseract \ | |
| --primary-engine-kwarg 'language=nep+eng' \ | |
| --primary-engine-kwarg 'psm=6' \ | |
| --secondary-engine-kwarg 'language=nep+eng' \ | |
| --secondary-engine-kwarg 'psm=13' | |
| ``` | |
| This writes: | |
| - `calibration.json` | |
| - `calibration.md` | |
| - `model-card.json` | |
| The generated model card records calibration provenance, including the calibration experiment, benchmark report, and calibration eval manifest hash. `preflight-claim` and `review-claim` use that provenance to reject tuned-on-eval routed candidates. | |
| ## Multi-Variant Tesseract Ensemble | |
| For a richer Tesseract-only ensemble, calibrate several PSM variants from separate experiment artifacts: | |
| ```bash | |
| ocrtech calibrate-tesseract-psm-ensemble \ | |
| --primary-experiment-report outputs/psm13-calibration/experiment.json \ | |
| --variant-experiment psm7=outputs/psm7-calibration/experiment.json \ | |
| --variant-experiment psm11=outputs/psm11-calibration/experiment.json \ | |
| --variant-experiment psm13=outputs/psm13-calibration/experiment.json \ | |
| --out outputs/psm-ensemble-calibration \ | |
| --model-id ocrtech-tesseract-psm-ensemble | |
| ``` | |
| This writes a `quality_ranked_ensemble` model card with: | |
| - `primary_engine` | |
| - `alternative_engines` | |
| - per-variant `quality_bias` | |
| - calibration provenance | |
| Current evidence says this path is not promising on the mixed Nepali/English line pack: it improved on the calibration half and regressed on the untouched holdout. Keep it as a research tool, not as a recommended candidate. | |
| ## Script-Routed Tesseract + Surya | |
| For a script-aware routed candidate, calibrate a threshold over the primary engine output text: | |
| ```bash | |
| ocrtech calibrate-script-router \ | |
| --experiment-report outputs/surya-calibration/experiment.json \ | |
| --out outputs/script-router-calibration \ | |
| --model-id ocrtech-tesseract-surya-script-router \ | |
| --primary-engine tesseract \ | |
| --secondary-engine surya \ | |
| --primary-engine-kwarg 'language=nep+eng' \ | |
| --primary-engine-kwarg 'psm=6' \ | |
| --secondary-engine-kwarg 'mode=auto' | |
| ``` | |
| This writes a `script_select_composite` model card with: | |
| - `primary_engine` | |
| - `secondary_engine` | |
| - `script` | |
| - `primary_script_threshold` | |
| - optional `routing_granularity` (`document` by default, `page` for per-page routing) | |
| - optional `secondary_structure_backfill` to keep primary text selection while filling missing tables/figures from the secondary engine | |
| - calibration provenance | |
| Current live evidence on `k2` is better than the quality-score-only Tesseract router: calibration on the `89`-sample split selected `primary_script_threshold=0.4`, and the untouched `91`-sample holdout improved overall CER from `0.342234` to `0.313084` with `9` wins, `0` losses, and `82` ties against raw Tesseract. | |
| An exploratory `routing_granularity=page` run on that same holdout landed on the same numbers. That does not mean page routing is useless; it means this particular holdout is mostly single-page line-crop material. Page routing needs a true multi-page document eval pack before it can show a different outcome. | |
| On the newer rendered multi-page image-pack smoke, `secondary_structure_backfill=true` turned out to be useful. The document-level router still kept Tesseract text on the multi-page report page, but it backfilled Surya's table structure on that page and moved `table_cell_f1` from `0.0` to `1.0` on the rendered smoke pack without changing CER. | |
| That behavior survived on the broader rendered claim-ready image pack as well. The calibrated backfill model card `ocrtech-tesseract-surya-script-router-backfill-calibrated` kept `routing_granularity=document`, `secondary_structure_backfill=true`, and `primary_script_threshold=0.4`. On the `25`-sample rendered pack it still beat raw Tesseract on overall CER (`0.248654` vs `0.689214`) and kept a large structure advantage (`table_cell_f1` `0.955556` vs `0.0`). The weak spots are still visible, which is exactly why this should be treated as an engineering baseline rather than a claim candidate: `form` CER regressed slightly and aggregate reading-order accuracy on that pack is still lower than Tesseract. | |
| ## Table Cell Refinement Composite | |
| `table_cell_refine_composite` uses one engine for document/table structure and a second engine for crop OCR over detected table cells: | |
| ```bash | |
| ocrtech create-model-card \ | |
| --out outputs/table-cell-refine-empty-v1/model-card.json \ | |
| --model-id ocrtech-surya-tesseract-table-cell-refine-empty-v1 \ | |
| --backend table_cell_refine_composite \ | |
| --base-model surya+tesseract \ | |
| --backend-kwarg structure_engine=surya \ | |
| --backend-kwarg crop_engine=tesseract \ | |
| --backend-kwarg crop_padding=4 \ | |
| --backend-kwarg replacement_policy=empty | |
| ``` | |
| Policies: | |
| - `replacement_policy=empty`: only OCR empty table cells. This is the safe default. | |
| - `replacement_policy=always`: reread every detected table cell. This is experimental and can damage already-correct Surya cells. | |
| - `replacement_policy=guarded`: replace non-empty cells only when the crop engine returns a higher-confidence result by at least `min_replacement_confidence_delta`. | |
| - `deduplicate_tables=true`: drop overlapping duplicate tables after refinement. The parser also avoids re-inferring a table from text lines when an engine already supplied an overlapping structured table. | |
| Guarded skips and crop errors are recorded on the affected table cell's | |
| `metadata` in `document.json` when the reason is cell-specific. This makes table | |
| refinement failures inspectable without reverse-engineering aggregate table | |
| counts. | |
| Current hard15 evidence says the idea is not enough: | |
| - unsafe `always` policy tied Surya on text CER but regressed table-cell F1 from `0.979167` to `0.593750` | |
| - safe `empty` policy tied Surya exactly on hard15 CER, table-cell F1, and reading order | |
| - safe `empty` plus table de-duplication also tied Surya exactly on hard15 CER, table-cell F1, and reading order | |
| - hard15 validation status for the safe policies is `not_validated` | |
| So this backend is useful infrastructure for future missing-cell cases, not a claim candidate yet. | |
| ## Figure Caption Refinement Composite | |
| `figure_caption_refine_composite` keeps a primary document engine for layout and OCR, then uses a second engine only to replace nearby figure-caption lines when the replacement overlaps spatially and has a better caption-prefix score: | |
| ```bash | |
| ocrtech create-model-card \ | |
| --out outputs/figure-caption-refine-v1/model-card.json \ | |
| --model-id ocrtech-surya-tesseract-figure-caption-refine-v1 \ | |
| --backend figure_caption_refine_composite \ | |
| --base-model surya+tesseract \ | |
| --backend-kwarg primary_engine=surya \ | |
| --backend-kwarg caption_engine=tesseract \ | |
| --backend-kwarg min_caption_iou=0.5 \ | |
| --backend-kwarg min_score_delta=1 | |
| ``` | |
| The replacement preserves the primary engine's bbox and layout ordering. Only the caption text/confidence changes. This avoids the skew failure where a secondary OCR bbox overlaps the figure marker and prevents caption attachment. | |
| Current hard15 evidence: | |
| - figure-caption CER improved over Surya from `0.092323` to `0.052478` | |
| - figure detection F1 stayed `1.000000` | |
| - table-cell F1 stayed `0.979167` | |
| - reading-order pair accuracy stayed `0.520000` | |
| - hard15 validation status is still `not_validated` because broad CER/table gates do not pass | |
| An opt-in `correct_structural_labels=true` branch adds edit-distance guarded correction for common Nepali document-role labels such as `वार्षिक प्रतिवेदन`, `चित्र`, and `निष्कर्ष`. On hard15 this stronger branch improved: | |
| - overall hard_eval CER over Surya from `0.236839` to `0.204610` | |
| - figure CER from `0.091927` to `0.022866` | |
| - figure-caption CER from `0.092323` to `0.008746` | |
| Validation still reports `not_validated` because table CER and table-cell F1 tie Surya, but the overall CER gate and both figure gates pass. | |
| A stronger auditable variant keeps the same backend but enables: | |
| - `correct_structural_labels=true`, now covering `वार्षिक प्रतिवेदन`, `दैनिक हिसाब`, `चित्र`, and `निष्कर्ष` | |
| - `table_name_lexicon_path=configs/nepali-table-name-lexicon-v1.txt` | |
| - `table_name_max_distance=2` | |
| The table-name lexicon correction is intentionally narrow: it only applies to cells under a header close to `नाम`, and only when there is a unique nearest lexicon entry within the configured edit-distance threshold. The current lexicon is tiny and curated, so paper claims must describe this as lexicon-assisted structured extraction, not a general language model. | |
| The public-table variant also enables: | |
| - `normalize_table_grids=true` | |
| This removes empty columns, separator-only columns such as `—`, and exact | |
| duplicate adjacent columns from predicted tables. The rule is runtime-only and | |
| reference-free. It improved the 5-page public OmniDocBench smoke over raw Surya | |
| on table metrics, but the public validation still remains `not_validated` | |
| because the sample count and confidence gates are not met. | |
| The later `normalize-rows-v3` public-table model card keeps the same option and | |
| adds sparse bullet row-fragment merging inside the grid normalizer. This is also | |
| runtime-only and reference-free. On the same 5-page public OmniDocBench smoke it | |
| raises table metrics further (`table_cell_f1` `0.564335`, | |
| `table_teds_structure` `0.942836`, `table_teds_text` `0.675633`) and changes the | |
| `omnidocbench-00208...` fragmented table from `16x5` to `3x5`. It is not a | |
| validated model: text CER is slightly worse than Surya on that smoke, and the | |
| public validation gates still fail on sample count and bootstrap CI. | |
| On the larger public OmniDocBench `table40` slice, the same model has higher | |
| mean table metrics than Surya but does not validate: | |
| - CER `0.713326` versus Surya `0.713960` | |
| - table-cell F1 `0.323122` versus Surya `0.280062`, but `7` wins / `9` losses / | |
| `24` ties and CI low `-0.019152` | |
| - table TEDS structure `0.520959` versus Surya `0.503603`, but CI low | |
| `-0.038381` | |
| - table TEDS text `0.361666` versus Surya `0.358057`, below the configured | |
| margin | |
| - cell exactness `3440/10314` versus Surya `3421/10314` | |
| This points to the next model requirement: better table segmentation and | |
| large/wide-table handling. The current grid postprocessors are useful but too | |
| local to support a public SOTA claim. | |
| The `normalize-width-guard-v4` model card adds one guard to the same runtime | |
| postprocessor: tables wider than `12` predicted columns skip grid normalization. | |
| This directly addresses public table40 failures where v3 merged `15` sparse rows | |
| or dropped `26` columns in wide tables. On full public table40, v4 improves over | |
| v3 and passes two configured public table gates against Surya: | |
| - CER `0.707864` versus Surya `0.713960` | |
| - table-cell F1 `0.339319` versus Surya `0.280062`, CI low `0.001433` | |
| - table TEDS structure `0.545375` versus Surya `0.503603`, CI low `0.004790` | |
| - table TEDS text `0.386013` versus Surya `0.358057`, but CI low `-0.000220` | |
| - cell exactness `3453/10314` versus Surya `3421/10314` | |
| This is a better public table parser candidate than v3, but still not a SOTA | |
| engine: the text-sensitive table gate and broad public document gate remain open. | |
| On hard15 v3, this model reaches: | |
| - overall CER `0.010671` versus Surya `0.236839` | |
| - table-slice CER `0.000000` versus Surya `0.363636` | |
| - figure-caption CER `0.008746` versus Surya `0.092323` | |
| - table-cell F1 `1.000000` versus Surya `0.979167` | |
| - table exact cells `96/96` versus Surya `94/96` | |
| Configured validation is still `not_validated` because the table-cell F1 bootstrap CI low is `0.000000` on the tiny 8-page table slice. This is a sample-size limitation of the current gate, not permission to claim table SOTA. Build a larger independent table pack before making that claim. | |
| The table-name lexicon matcher now also supports suffix-preserving generated IDs such as `करिण-302` -> `किरण-302`. The correction still only fires in columns whose header is close to `नाम`, and still requires a unique nearest base-name match within the configured edit distance. | |
| Model-card audit validates the caption/table-refine thresholds and verifies | |
| `table_name_lexicon_path` resolves to a real file. Treat a missing lexicon or | |
| out-of-range parser knob as a failed artifact, not a runtime surprise inside a | |
| benchmark. | |
| On the larger `60`-sample table-ledger pack, this same model passes the targeted Surya table validation: | |
| - table-cell F1: candidate `1.000000`, Surya `0.979167`, CI low `0.012500` | |
| - table exact match: candidate `1.000000`, Surya `0.750000`, CI low `0.150000` | |
| - table structure similarity: candidate `1.000000`, Surya `1.000000`, non-regression guard passes | |
| - cell exactness: candidate `720/720`, Surya `705/720` | |
| This is valid evidence for lexicon-assisted table-ledger cell-content extraction with grid-structure non-regression. It is not a general OCR SOTA result, and it does not prove improvement on complex merged-cell tables. | |
| On the first `complex-table-ledger` run, the same candidate improved cell-content F1 over Surya (`0.928333` vs `0.915000`) after extending name correction below grouped headers, but it did not improve merged-header structure. | |
| The later span-inference parser branch adds conservative repeated-header colspan inference. On the same generated complex-table pack, it reaches: | |
| - table-cell F1 `0.995000` versus Surya `0.915000` | |
| - table exact match `0.925000` versus Surya `0.000000` | |
| - span-aware table structure similarity `1.000000` versus Surya `0.875000` | |
| - TEDS-style table tree structure similarity `1.000000` versus Surya `0.904762` | |
| - TEDS-style text-sensitive table tree similarity `0.996250` versus Surya `0.891667` | |
| - cell exactness `597/600` versus Surya `549/600` | |
| This is a parser/postprocessing improvement for a generated repeated-header pattern. It is not a trained merged-cell model and does not support a broad complex-table SOTA claim without real scanned tables and varied span structures. | |
| The public OmniDocBench table40 width-guard model card is: | |
| - `outputs/figure-caption-table-lexicon-normalize-width-guard-v4/model-card.json` | |
| - model id: `ocrtech-surya-tesseract-figure-table-lexicon-normalize-width-guard-v4` | |
| - audit: `outputs/audit-figure-caption-table-lexicon-normalize-width-guard-v4-model/model-audit.json` | |
| This candidate keeps the normalized table-grid postprocessors but skips grid | |
| normalization when predicted table width is greater than `12`. On the public | |
| OmniDocBench table40 combined run, it improves over Surya on table-cell F1 | |
| (`0.339319` vs `0.280062`) and structure-only TEDS (`0.545375` vs `0.503603`), | |
| but the run remains `not_validated` because the public document CER gate is | |
| underpowered and table TEDS text is borderline in the separate-run comparison. | |
| For postprocessor-only evidence, use | |
| `outputs/experiment-public-omnidocbench-table40-derived-postprocess-v1/benchmark/report.json`. | |
| That derived benchmark applies the table postprocessors to the exact same Surya | |
| `document.json` outputs, so CER stays identical while table metrics isolate the | |
| postprocessor contribution. It passes the configured table-cell F1, | |
| structure-only TEDS, and text-sensitive TEDS gates on `40` pages, but it is a | |
| validation lens rather than a deployable model card. | |
| ## Audit | |
| ```bash | |
| ocrtech audit-model runs/recognizer/package/model-card.json \ | |
| --out outputs/model-audit | |
| ``` | |
| The audit verifies: | |
| - required artifact paths exist | |
| - recorded SHA-256 values match for file artifacts | |
| - recorded `size_bytes` values match for file and directory artifacts | |
| - backend is supported | |
| - the card shape is valid | |
| `audit-model` does not by itself prove claim readiness. For `quality_select_composite`, `quality_ranked_ensemble`, and `script_select_composite`, the calibration provenance must also be separate from the final claim eval manifest. | |
| For recognizer branch admission, attach the hard-gate validation report when | |
| creating or packaging a model card: | |
| ```bash | |
| ocrtech create-model-card \ | |
| --out runs/recognizer/package/model-card.json \ | |
| --model-id ocrtech-next-recognizer \ | |
| --backend paddleocr \ | |
| --artifact runs/recognizer/export/inference/inference.pdiparams \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --eval-manifest data/splits/eval.jsonl \ | |
| --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json | |
| ``` | |
| Claim-grade preflight can then require the admission gate with | |
| `--require-model-admission`. Admission reports must include quantitative | |
| passing gate evidence; a bare `claim_status=validated` report is rejected. | |
| ## Parse With Candidate | |
| ```bash | |
| ocrtech parse-doc page.png \ | |
| --engine candidate \ | |
| --model-config runs/recognizer/package/model-card.json \ | |
| --fallback-model-config runs/vlm-fallback/package/model-card.json \ | |
| --fallback-min-quality-score 0.70 \ | |
| --out outputs/page | |
| ``` | |
| When a fallback engine or fallback model card is configured, the parser does not | |
| silently swap engines. It records `metadata.fallback` in `document.json` with | |
| the primary engine, fallback engine/model config, average primary confidence, | |
| confidence threshold, optional primary quality score and quality threshold, | |
| trigger reason, and outcome. This is required for paper evidence: a selective | |
| VLM/document fallback must report when it actually fired and whether it | |
| succeeded. The fallback can be triggered by low OCR confidence or, when | |
| `--fallback-min-quality-score` is set, by poor parsed-document quality. | |
| ## Benchmark With Candidate | |
| ```bash | |
| ocrtech run-experiment \ | |
| --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ | |
| --references-dir data/eval-pack/references \ | |
| --candidate-model-config runs/recognizer/package/model-card.json \ | |
| --fallback-model-config runs/vlm-fallback/package/model-card.json \ | |
| --low-confidence-threshold 0.80 \ | |
| --fallback-min-quality-score 0.70 \ | |
| --baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \ | |
| --validation-config validation.json \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --out outputs/claim-experiment \ | |
| --require-validated | |
| ``` | |
| If `--candidate-model-config` is omitted, `ours` uses the local `auto` engine for smoke tests. That is not valid SOTA evidence. | |
| Benchmark and experiment fallback flags apply only to the candidate/ours run. | |
| They are meant to measure the deployed hybrid engine, not to contaminate | |
| baseline rows. `experiment.json` records `fallback_engine`, | |
| `fallback_model_config`, `fallback_model_config_sha256`, and | |
| `low_confidence_threshold`, plus `fallback_min_quality_score` when configured, | |
| so claim reviewers can tell which routing policy was actually evaluated. | |