ocr-tech-k2 / docs /MODELS.md
prosperious's picture
Add 51 files (.gitignore ...)
1c43956 verified
|
Raw History Blame Contribute Delete
24.4 kB
# Candidate Model Cards
The candidate OCR engine must run from an explicit model card. Do not benchmark `ours` against external systems unless the candidate model card is attached to the experiment.
Supported candidate backends:
- `paddleocr`
- `hf_vision_encoder_decoder`
- `sidecar`
- `tesseract`
- `text_correction_composite`
- `quality_select_composite`
- `quality_ranked_ensemble`
- `script_select_composite`
- `line_align_composite`
- `table_cell_refine_composite`
## Export And Package
After `train-recognizer` finishes and a PaddleOCR checkpoint exists, export an inference model:
```bash
ocrtech export-recognizer \
--training-config runs/recognizer/paddle_recognizer.yml \
--checkpoint runs/recognizer/checkpoints/best_accuracy \
--out runs/recognizer/export
```
Run the generated export command from a PaddleOCR checkout, or pass `--run` when the checkout exists locally.
Then package the exported inference directory into an auditable candidate model:
```bash
ocrtech package-recognizer \
--inference-dir runs/recognizer/export/inference \
--dictionary runs/recognizer/nep_eng_dict.txt \
--out runs/recognizer/package \
--model-id ocrtech-nep-eng-rec-v1 \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--training-summary runs/recognizer/recognizer-training-summary.json \
--admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
```
This writes `runs/recognizer/package/model-card.json` and copies all required inference artifacts under `runs/recognizer/package/artifacts/`.
## Training Recipe Evidence
`train-recognizer` writes a machine-readable
`recognizer-training-summary.json` beside the backend training config:
`paddle_recognizer.yml` for PaddleOCR and `hf_recognizer_config.json` for
Hugging Face vision-encoder-decoder runs. Preserve it with the recipe, bundle,
training logs, exported inference model, and packaged model card. It records:
- train/eval manifest paths and SHA-256 hashes
- train/eval sample counts, dataset counts, and slice counts
- generated label, dictionary, and config hashes
- dictionary character coverage, including ASCII and Devanagari presence
- the recipe hyperparameters used for launch
When a custom `--dictionary` is supplied, recipe generation now checks it
against the exact normalized PaddleOCR label text for every train and eval
sample. Missing characters fail before a remote training job is packaged. This
prevents a recognizer run from silently launching with labels that the
recognition head cannot emit.
This summary is not a quality result. It is launch provenance for the local
recognizer branch, and it helps distinguish a reproducible recognizer attempt
from an undocumented remote training run.
`bundle-recognizer-training` includes this summary in the archive and records
its path and SHA-256 under `training_summary` in
`recognizer-training-bundle.json`. Check that field after syncing the bundle to
a CUDA host.
For real GPU handoffs, pass the latest recognizer failure review to both recipe
generation and bundling. This rejects families already stopped by admission
evidence before another remote job is launched:
```bash
ocrtech train-recognizer \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--out runs/recognizer \
--failure-review outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json
ocrtech bundle-recognizer-training \
--recipe-dir runs/recognizer \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--out runs/recognizer-training-bundle \
--require-corpus-audit \
--failure-review outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json
```
If that review contains `stop_family` for the recipe backend/base model, the
command fails with the rejected-run and win/loss/tie evidence. Changing the run
name, learning rate, or corpus is not enough to bypass a stopped family; use a
materially different base recognizer, objective, or document-native model.
Pass the same file to `package-recognizer --training-summary` or
`package-hf-recognizer --training-summary` so the final `model-card.json`
records the summary path and SHA-256 under `provenance.training_summary`.
## Hugging Face Packaging
For an arm64-compatible recognizer path, train a Hugging Face vision-text model and package the saved model directory:
```bash
ocrtech train-recognizer \
--backend hf-vision-encoder-decoder \
--base-model microsoft/trocr-small-printed \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--out runs/hf-recognizer
ocrtech package-hf-recognizer \
--model-dir runs/hf-recognizer/final-model \
--out runs/hf-recognizer/package \
--model-id ocrtech-hf-nep-eng-rec-v1 \
--base-model microsoft/trocr-small-printed \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--training-summary runs/hf-recognizer/recognizer-training-summary.json \
--admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
```
The HF recipe writes `runs/hf-recognizer/recognizer-training-summary.json` at
recipe-generation time. That summary records train/eval manifest hashes,
script coverage, config hash, and hyperparameters. Packaging should include it
just like the PaddleOCR path; otherwise the trained-recognizer provenance gate
will have weaker launch evidence.
This writes `runs/hf-recognizer/package/model-card.json` and records backend kwargs for loading the saved processor and model directory.
## Create Manually
```bash
ocrtech create-model-card \
--out runs/recognizer/model-card.json \
--model-id ocrtech-nep-eng-rec-v1 \
--backend paddleocr \
--base-model PaddlePaddle/devanagari_PP-OCRv3_mobile_rec \
--artifact runs/recognizer/checkpoints/best_accuracy.pdparams \
--artifact runs/recognizer/nep_eng_dict.txt \
--paddleocr-kwarg lang=devanagari \
--paddleocr-kwarg rec_model_dir=runs/recognizer/inference \
--paddleocr-kwarg rec_char_dict_path=runs/recognizer/nep_eng_dict.txt \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--metrics-report outputs/claim-experiment/validation/validation.json
```
The backend kwargs are passed through to the selected runtime backend. For `paddleocr`, keep them explicit in the model card because PaddleOCR constructor names can change by version.
## Routed Candidates
For routed or thresholded candidates, do not hand-edit the threshold into a model card and call that evidence. Calibrate it from a separate experiment artifact:
```bash
ocrtech calibrate-quality-router \
--experiment-report outputs/tesseract-psm13-calibration/experiment.json \
--out outputs/quality-router-calibration \
--model-id ocrtech-tesseract-quality-router \
--primary-engine tesseract \
--secondary-engine tesseract \
--primary-engine-kwarg 'language=nep+eng' \
--primary-engine-kwarg 'psm=6' \
--secondary-engine-kwarg 'language=nep+eng' \
--secondary-engine-kwarg 'psm=13'
```
This writes:
- `calibration.json`
- `calibration.md`
- `model-card.json`
The generated model card records calibration provenance, including the calibration experiment, benchmark report, and calibration eval manifest hash. `preflight-claim` and `review-claim` use that provenance to reject tuned-on-eval routed candidates.
## Multi-Variant Tesseract Ensemble
For a richer Tesseract-only ensemble, calibrate several PSM variants from separate experiment artifacts:
```bash
ocrtech calibrate-tesseract-psm-ensemble \
--primary-experiment-report outputs/psm13-calibration/experiment.json \
--variant-experiment psm7=outputs/psm7-calibration/experiment.json \
--variant-experiment psm11=outputs/psm11-calibration/experiment.json \
--variant-experiment psm13=outputs/psm13-calibration/experiment.json \
--out outputs/psm-ensemble-calibration \
--model-id ocrtech-tesseract-psm-ensemble
```
This writes a `quality_ranked_ensemble` model card with:
- `primary_engine`
- `alternative_engines`
- per-variant `quality_bias`
- calibration provenance
Current evidence says this path is not promising on the mixed Nepali/English line pack: it improved on the calibration half and regressed on the untouched holdout. Keep it as a research tool, not as a recommended candidate.
## Script-Routed Tesseract + Surya
For a script-aware routed candidate, calibrate a threshold over the primary engine output text:
```bash
ocrtech calibrate-script-router \
--experiment-report outputs/surya-calibration/experiment.json \
--out outputs/script-router-calibration \
--model-id ocrtech-tesseract-surya-script-router \
--primary-engine tesseract \
--secondary-engine surya \
--primary-engine-kwarg 'language=nep+eng' \
--primary-engine-kwarg 'psm=6' \
--secondary-engine-kwarg 'mode=auto'
```
This writes a `script_select_composite` model card with:
- `primary_engine`
- `secondary_engine`
- `script`
- `primary_script_threshold`
- optional `routing_granularity` (`document` by default, `page` for per-page routing)
- optional `secondary_structure_backfill` to keep primary text selection while filling missing tables/figures from the secondary engine
- calibration provenance
Current live evidence on `k2` is better than the quality-score-only Tesseract router: calibration on the `89`-sample split selected `primary_script_threshold=0.4`, and the untouched `91`-sample holdout improved overall CER from `0.342234` to `0.313084` with `9` wins, `0` losses, and `82` ties against raw Tesseract.
An exploratory `routing_granularity=page` run on that same holdout landed on the same numbers. That does not mean page routing is useless; it means this particular holdout is mostly single-page line-crop material. Page routing needs a true multi-page document eval pack before it can show a different outcome.
On the newer rendered multi-page image-pack smoke, `secondary_structure_backfill=true` turned out to be useful. The document-level router still kept Tesseract text on the multi-page report page, but it backfilled Surya's table structure on that page and moved `table_cell_f1` from `0.0` to `1.0` on the rendered smoke pack without changing CER.
That behavior survived on the broader rendered claim-ready image pack as well. The calibrated backfill model card `ocrtech-tesseract-surya-script-router-backfill-calibrated` kept `routing_granularity=document`, `secondary_structure_backfill=true`, and `primary_script_threshold=0.4`. On the `25`-sample rendered pack it still beat raw Tesseract on overall CER (`0.248654` vs `0.689214`) and kept a large structure advantage (`table_cell_f1` `0.955556` vs `0.0`). The weak spots are still visible, which is exactly why this should be treated as an engineering baseline rather than a claim candidate: `form` CER regressed slightly and aggregate reading-order accuracy on that pack is still lower than Tesseract.
## Table Cell Refinement Composite
`table_cell_refine_composite` uses one engine for document/table structure and a second engine for crop OCR over detected table cells:
```bash
ocrtech create-model-card \
--out outputs/table-cell-refine-empty-v1/model-card.json \
--model-id ocrtech-surya-tesseract-table-cell-refine-empty-v1 \
--backend table_cell_refine_composite \
--base-model surya+tesseract \
--backend-kwarg structure_engine=surya \
--backend-kwarg crop_engine=tesseract \
--backend-kwarg crop_padding=4 \
--backend-kwarg replacement_policy=empty
```
Policies:
- `replacement_policy=empty`: only OCR empty table cells. This is the safe default.
- `replacement_policy=always`: reread every detected table cell. This is experimental and can damage already-correct Surya cells.
- `deduplicate_tables=true`: drop overlapping duplicate tables after refinement. The parser also avoids re-inferring a table from text lines when an engine already supplied an overlapping structured table.
Current hard15 evidence says the idea is not enough:
- unsafe `always` policy tied Surya on text CER but regressed table-cell F1 from `0.979167` to `0.593750`
- safe `empty` policy tied Surya exactly on hard15 CER, table-cell F1, and reading order
- safe `empty` plus table de-duplication also tied Surya exactly on hard15 CER, table-cell F1, and reading order
- hard15 validation status for the safe policies is `not_validated`
So this backend is useful infrastructure for future missing-cell cases, not a claim candidate yet.
## Figure Caption Refinement Composite
`figure_caption_refine_composite` keeps a primary document engine for layout and OCR, then uses a second engine only to replace nearby figure-caption lines when the replacement overlaps spatially and has a better caption-prefix score:
```bash
ocrtech create-model-card \
--out outputs/figure-caption-refine-v1/model-card.json \
--model-id ocrtech-surya-tesseract-figure-caption-refine-v1 \
--backend figure_caption_refine_composite \
--base-model surya+tesseract \
--backend-kwarg primary_engine=surya \
--backend-kwarg caption_engine=tesseract \
--backend-kwarg min_caption_iou=0.5 \
--backend-kwarg min_score_delta=1
```
The replacement preserves the primary engine's bbox and layout ordering. Only the caption text/confidence changes. This avoids the skew failure where a secondary OCR bbox overlaps the figure marker and prevents caption attachment.
Current hard15 evidence:
- figure-caption CER improved over Surya from `0.092323` to `0.052478`
- figure detection F1 stayed `1.000000`
- table-cell F1 stayed `0.979167`
- reading-order pair accuracy stayed `0.520000`
- hard15 validation status is still `not_validated` because broad CER/table gates do not pass
An opt-in `correct_structural_labels=true` branch adds edit-distance guarded correction for common Nepali document-role labels such as `वार्षिक प्रतिवेदन`, `चित्र`, and `निष्कर्ष`. On hard15 this stronger branch improved:
- overall hard_eval CER over Surya from `0.236839` to `0.204610`
- figure CER from `0.091927` to `0.022866`
- figure-caption CER from `0.092323` to `0.008746`
Validation still reports `not_validated` because table CER and table-cell F1 tie Surya, but the overall CER gate and both figure gates pass.
A stronger auditable variant keeps the same backend but enables:
- `correct_structural_labels=true`, now covering `वार्षिक प्रतिवेदन`, `दैनिक हिसाब`, `चित्र`, and `निष्कर्ष`
- `table_name_lexicon_path=configs/nepali-table-name-lexicon-v1.txt`
- `table_name_max_distance=2`
The table-name lexicon correction is intentionally narrow: it only applies to cells under a header close to `नाम`, and only when there is a unique nearest lexicon entry within the configured edit-distance threshold. The current lexicon is tiny and curated, so paper claims must describe this as lexicon-assisted structured extraction, not a general language model.
The public-table variant also enables:
- `normalize_table_grids=true`
This removes empty columns, separator-only columns such as `—`, and exact
duplicate adjacent columns from predicted tables. The rule is runtime-only and
reference-free. It improved the 5-page public OmniDocBench smoke over raw Surya
on table metrics, but the public validation still remains `not_validated`
because the sample count and confidence gates are not met.
The later `normalize-rows-v3` public-table model card keeps the same option and
adds sparse bullet row-fragment merging inside the grid normalizer. This is also
runtime-only and reference-free. On the same 5-page public OmniDocBench smoke it
raises table metrics further (`table_cell_f1` `0.564335`,
`table_teds_structure` `0.942836`, `table_teds_text` `0.675633`) and changes the
`omnidocbench-00208...` fragmented table from `16x5` to `3x5`. It is not a
validated model: text CER is slightly worse than Surya on that smoke, and the
public validation gates still fail on sample count and bootstrap CI.
On the larger public OmniDocBench `table40` slice, the same model has higher
mean table metrics than Surya but does not validate:
- CER `0.713326` versus Surya `0.713960`
- table-cell F1 `0.323122` versus Surya `0.280062`, but `7` wins / `9` losses /
`24` ties and CI low `-0.019152`
- table TEDS structure `0.520959` versus Surya `0.503603`, but CI low
`-0.038381`
- table TEDS text `0.361666` versus Surya `0.358057`, below the configured
margin
- cell exactness `3440/10314` versus Surya `3421/10314`
This points to the next model requirement: better table segmentation and
large/wide-table handling. The current grid postprocessors are useful but too
local to support a public SOTA claim.
The `normalize-width-guard-v4` model card adds one guard to the same runtime
postprocessor: tables wider than `12` predicted columns skip grid normalization.
This directly addresses public table40 failures where v3 merged `15` sparse rows
or dropped `26` columns in wide tables. On full public table40, v4 improves over
v3 and passes two configured public table gates against Surya:
- CER `0.707864` versus Surya `0.713960`
- table-cell F1 `0.339319` versus Surya `0.280062`, CI low `0.001433`
- table TEDS structure `0.545375` versus Surya `0.503603`, CI low `0.004790`
- table TEDS text `0.386013` versus Surya `0.358057`, but CI low `-0.000220`
- cell exactness `3453/10314` versus Surya `3421/10314`
This is a better public table parser candidate than v3, but still not a SOTA
engine: the text-sensitive table gate and broad public document gate remain open.
On hard15 v3, this model reaches:
- overall CER `0.010671` versus Surya `0.236839`
- table-slice CER `0.000000` versus Surya `0.363636`
- figure-caption CER `0.008746` versus Surya `0.092323`
- table-cell F1 `1.000000` versus Surya `0.979167`
- table exact cells `96/96` versus Surya `94/96`
Configured validation is still `not_validated` because the table-cell F1 bootstrap CI low is `0.000000` on the tiny 8-page table slice. This is a sample-size limitation of the current gate, not permission to claim table SOTA. Build a larger independent table pack before making that claim.
The table-name lexicon matcher now also supports suffix-preserving generated IDs such as `करिण-302` -> `किरण-302`. The correction still only fires in columns whose header is close to `नाम`, and still requires a unique nearest base-name match within the configured edit distance.
On the larger `60`-sample table-ledger pack, this same model passes the targeted Surya table validation:
- table-cell F1: candidate `1.000000`, Surya `0.979167`, CI low `0.012500`
- table exact match: candidate `1.000000`, Surya `0.750000`, CI low `0.150000`
- table structure similarity: candidate `1.000000`, Surya `1.000000`, non-regression guard passes
- cell exactness: candidate `720/720`, Surya `705/720`
This is valid evidence for lexicon-assisted table-ledger cell-content extraction with grid-structure non-regression. It is not a general OCR SOTA result, and it does not prove improvement on complex merged-cell tables.
On the first `complex-table-ledger` run, the same candidate improved cell-content F1 over Surya (`0.928333` vs `0.915000`) after extending name correction below grouped headers, but it did not improve merged-header structure.
The later span-inference parser branch adds conservative repeated-header colspan inference. On the same generated complex-table pack, it reaches:
- table-cell F1 `0.995000` versus Surya `0.915000`
- table exact match `0.925000` versus Surya `0.000000`
- span-aware table structure similarity `1.000000` versus Surya `0.875000`
- TEDS-style table tree structure similarity `1.000000` versus Surya `0.904762`
- TEDS-style text-sensitive table tree similarity `0.996250` versus Surya `0.891667`
- cell exactness `597/600` versus Surya `549/600`
This is a parser/postprocessing improvement for a generated repeated-header pattern. It is not a trained merged-cell model and does not support a broad complex-table SOTA claim without real scanned tables and varied span structures.
The public OmniDocBench table40 width-guard model card is:
- `outputs/figure-caption-table-lexicon-normalize-width-guard-v4/model-card.json`
- model id: `ocrtech-surya-tesseract-figure-table-lexicon-normalize-width-guard-v4`
- audit: `outputs/audit-figure-caption-table-lexicon-normalize-width-guard-v4-model/model-audit.json`
This candidate keeps the normalized table-grid postprocessors but skips grid
normalization when predicted table width is greater than `12`. On the public
OmniDocBench table40 combined run, it improves over Surya on table-cell F1
(`0.339319` vs `0.280062`) and structure-only TEDS (`0.545375` vs `0.503603`),
but the run remains `not_validated` because the public document CER gate is
underpowered and table TEDS text is borderline in the separate-run comparison.
For postprocessor-only evidence, use
`outputs/experiment-public-omnidocbench-table40-derived-postprocess-v1/benchmark/report.json`.
That derived benchmark applies the table postprocessors to the exact same Surya
`document.json` outputs, so CER stays identical while table metrics isolate the
postprocessor contribution. It passes the configured table-cell F1,
structure-only TEDS, and text-sensitive TEDS gates on `40` pages, but it is a
validation lens rather than a deployable model card.
## Audit
```bash
ocrtech audit-model runs/recognizer/package/model-card.json \
--out outputs/model-audit
```
The audit verifies:
- required artifact paths exist
- recorded SHA-256 values match for file artifacts
- backend is supported
- the card shape is valid
`audit-model` does not by itself prove claim readiness. For `quality_select_composite`, `quality_ranked_ensemble`, and `script_select_composite`, the calibration provenance must also be separate from the final claim eval manifest.
For recognizer branch admission, attach the hard-gate validation report when
creating or packaging a model card:
```bash
ocrtech create-model-card \
--out runs/recognizer/package/model-card.json \
--model-id ocrtech-next-recognizer \
--backend paddleocr \
--artifact runs/recognizer/export/inference/inference.pdiparams \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
```
Claim-grade preflight can then require the admission gate with
`--require-model-admission`. Admission reports must include quantitative
passing gate evidence; a bare `claim_status=validated` report is rejected.
## Parse With Candidate
```bash
ocrtech parse-doc page.png \
--engine candidate \
--model-config runs/recognizer/package/model-card.json \
--fallback-model-config runs/vlm-fallback/package/model-card.json \
--out outputs/page
```
When a fallback engine or fallback model card is configured, the parser does not
silently swap engines. It records `metadata.fallback` in `document.json` with
the primary engine, fallback engine/model config, average primary confidence,
threshold, trigger reason, and outcome. This is required for paper evidence: a
selective VLM/document fallback must report when it actually fired and whether
it succeeded.
## Benchmark With Candidate
```bash
ocrtech run-experiment \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--candidate-model-config runs/recognizer/package/model-card.json \
--fallback-model-config runs/vlm-fallback/package/model-card.json \
--low-confidence-threshold 0.80 \
--baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
--validation-config validation.json \
--train-manifest data/splits/train.jsonl \
--out outputs/claim-experiment \
--require-validated
```
If `--candidate-model-config` is omitted, `ours` uses the local `auto` engine for smoke tests. That is not valid SOTA evidence.
Benchmark and experiment fallback flags apply only to the candidate/ours run.
They are meant to measure the deployed hybrid engine, not to contaminate
baseline rows. `experiment.json` records `fallback_engine`,
`fallback_model_config`, `fallback_model_config_sha256`, and
`low_confidence_threshold` so claim reviewers can tell which routing policy was
actually evaluated.