ocr-tech-k2 / docs /MODELS.md
prosperious's picture
Add 51 files (.gitignore ...)
1c43956 verified
|
Raw History Blame Contribute Delete
24.4 kB

Candidate Model Cards

The candidate OCR engine must run from an explicit model card. Do not benchmark ours against external systems unless the candidate model card is attached to the experiment.

Supported candidate backends:

  • paddleocr
  • hf_vision_encoder_decoder
  • sidecar
  • tesseract
  • text_correction_composite
  • quality_select_composite
  • quality_ranked_ensemble
  • script_select_composite
  • line_align_composite
  • table_cell_refine_composite

Export And Package

After train-recognizer finishes and a PaddleOCR checkpoint exists, export an inference model:

ocrtech export-recognizer \
  --training-config runs/recognizer/paddle_recognizer.yml \
  --checkpoint runs/recognizer/checkpoints/best_accuracy \
  --out runs/recognizer/export

Run the generated export command from a PaddleOCR checkout, or pass --run when the checkout exists locally.

Then package the exported inference directory into an auditable candidate model:

ocrtech package-recognizer \
  --inference-dir runs/recognizer/export/inference \
  --dictionary runs/recognizer/nep_eng_dict.txt \
  --out runs/recognizer/package \
  --model-id ocrtech-nep-eng-rec-v1 \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --training-summary runs/recognizer/recognizer-training-summary.json \
  --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json

This writes runs/recognizer/package/model-card.json and copies all required inference artifacts under runs/recognizer/package/artifacts/.

Training Recipe Evidence

train-recognizer writes a machine-readable recognizer-training-summary.json beside the backend training config: paddle_recognizer.yml for PaddleOCR and hf_recognizer_config.json for Hugging Face vision-encoder-decoder runs. Preserve it with the recipe, bundle, training logs, exported inference model, and packaged model card. It records:

  • train/eval manifest paths and SHA-256 hashes
  • train/eval sample counts, dataset counts, and slice counts
  • generated label, dictionary, and config hashes
  • dictionary character coverage, including ASCII and Devanagari presence
  • the recipe hyperparameters used for launch

When a custom --dictionary is supplied, recipe generation now checks it against the exact normalized PaddleOCR label text for every train and eval sample. Missing characters fail before a remote training job is packaged. This prevents a recognizer run from silently launching with labels that the recognition head cannot emit.

This summary is not a quality result. It is launch provenance for the local recognizer branch, and it helps distinguish a reproducible recognizer attempt from an undocumented remote training run.

bundle-recognizer-training includes this summary in the archive and records its path and SHA-256 under training_summary in recognizer-training-bundle.json. Check that field after syncing the bundle to a CUDA host.

For real GPU handoffs, pass the latest recognizer failure review to both recipe generation and bundling. This rejects families already stopped by admission evidence before another remote job is launched:

ocrtech train-recognizer \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --out runs/recognizer \
  --failure-review outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json

ocrtech bundle-recognizer-training \
  --recipe-dir runs/recognizer \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --out runs/recognizer-training-bundle \
  --require-corpus-audit \
  --failure-review outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json

If that review contains stop_family for the recipe backend/base model, the command fails with the rejected-run and win/loss/tie evidence. Changing the run name, learning rate, or corpus is not enough to bypass a stopped family; use a materially different base recognizer, objective, or document-native model.

Pass the same file to package-recognizer --training-summary or package-hf-recognizer --training-summary so the final model-card.json records the summary path and SHA-256 under provenance.training_summary.

Hugging Face Packaging

For an arm64-compatible recognizer path, train a Hugging Face vision-text model and package the saved model directory:

ocrtech train-recognizer \
  --backend hf-vision-encoder-decoder \
  --base-model microsoft/trocr-small-printed \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --out runs/hf-recognizer

ocrtech package-hf-recognizer \
  --model-dir runs/hf-recognizer/final-model \
  --out runs/hf-recognizer/package \
  --model-id ocrtech-hf-nep-eng-rec-v1 \
  --base-model microsoft/trocr-small-printed \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --training-summary runs/hf-recognizer/recognizer-training-summary.json \
  --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json

The HF recipe writes runs/hf-recognizer/recognizer-training-summary.json at recipe-generation time. That summary records train/eval manifest hashes, script coverage, config hash, and hyperparameters. Packaging should include it just like the PaddleOCR path; otherwise the trained-recognizer provenance gate will have weaker launch evidence.

This writes runs/hf-recognizer/package/model-card.json and records backend kwargs for loading the saved processor and model directory.

Create Manually

ocrtech create-model-card \
  --out runs/recognizer/model-card.json \
  --model-id ocrtech-nep-eng-rec-v1 \
  --backend paddleocr \
  --base-model PaddlePaddle/devanagari_PP-OCRv3_mobile_rec \
  --artifact runs/recognizer/checkpoints/best_accuracy.pdparams \
  --artifact runs/recognizer/nep_eng_dict.txt \
  --paddleocr-kwarg lang=devanagari \
  --paddleocr-kwarg rec_model_dir=runs/recognizer/inference \
  --paddleocr-kwarg rec_char_dict_path=runs/recognizer/nep_eng_dict.txt \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --metrics-report outputs/claim-experiment/validation/validation.json

The backend kwargs are passed through to the selected runtime backend. For paddleocr, keep them explicit in the model card because PaddleOCR constructor names can change by version.

Routed Candidates

For routed or thresholded candidates, do not hand-edit the threshold into a model card and call that evidence. Calibrate it from a separate experiment artifact:

ocrtech calibrate-quality-router \
  --experiment-report outputs/tesseract-psm13-calibration/experiment.json \
  --out outputs/quality-router-calibration \
  --model-id ocrtech-tesseract-quality-router \
  --primary-engine tesseract \
  --secondary-engine tesseract \
  --primary-engine-kwarg 'language=nep+eng' \
  --primary-engine-kwarg 'psm=6' \
  --secondary-engine-kwarg 'language=nep+eng' \
  --secondary-engine-kwarg 'psm=13'

This writes:

  • calibration.json
  • calibration.md
  • model-card.json

The generated model card records calibration provenance, including the calibration experiment, benchmark report, and calibration eval manifest hash. preflight-claim and review-claim use that provenance to reject tuned-on-eval routed candidates.

Multi-Variant Tesseract Ensemble

For a richer Tesseract-only ensemble, calibrate several PSM variants from separate experiment artifacts:

ocrtech calibrate-tesseract-psm-ensemble \
  --primary-experiment-report outputs/psm13-calibration/experiment.json \
  --variant-experiment psm7=outputs/psm7-calibration/experiment.json \
  --variant-experiment psm11=outputs/psm11-calibration/experiment.json \
  --variant-experiment psm13=outputs/psm13-calibration/experiment.json \
  --out outputs/psm-ensemble-calibration \
  --model-id ocrtech-tesseract-psm-ensemble

This writes a quality_ranked_ensemble model card with:

  • primary_engine
  • alternative_engines
  • per-variant quality_bias
  • calibration provenance

Current evidence says this path is not promising on the mixed Nepali/English line pack: it improved on the calibration half and regressed on the untouched holdout. Keep it as a research tool, not as a recommended candidate.

Script-Routed Tesseract + Surya

For a script-aware routed candidate, calibrate a threshold over the primary engine output text:

ocrtech calibrate-script-router \
  --experiment-report outputs/surya-calibration/experiment.json \
  --out outputs/script-router-calibration \
  --model-id ocrtech-tesseract-surya-script-router \
  --primary-engine tesseract \
  --secondary-engine surya \
  --primary-engine-kwarg 'language=nep+eng' \
  --primary-engine-kwarg 'psm=6' \
  --secondary-engine-kwarg 'mode=auto'

This writes a script_select_composite model card with:

  • primary_engine
  • secondary_engine
  • script
  • primary_script_threshold
  • optional routing_granularity (document by default, page for per-page routing)
  • optional secondary_structure_backfill to keep primary text selection while filling missing tables/figures from the secondary engine
  • calibration provenance

Current live evidence on k2 is better than the quality-score-only Tesseract router: calibration on the 89-sample split selected primary_script_threshold=0.4, and the untouched 91-sample holdout improved overall CER from 0.342234 to 0.313084 with 9 wins, 0 losses, and 82 ties against raw Tesseract.

An exploratory routing_granularity=page run on that same holdout landed on the same numbers. That does not mean page routing is useless; it means this particular holdout is mostly single-page line-crop material. Page routing needs a true multi-page document eval pack before it can show a different outcome.

On the newer rendered multi-page image-pack smoke, secondary_structure_backfill=true turned out to be useful. The document-level router still kept Tesseract text on the multi-page report page, but it backfilled Surya's table structure on that page and moved table_cell_f1 from 0.0 to 1.0 on the rendered smoke pack without changing CER.

That behavior survived on the broader rendered claim-ready image pack as well. The calibrated backfill model card ocrtech-tesseract-surya-script-router-backfill-calibrated kept routing_granularity=document, secondary_structure_backfill=true, and primary_script_threshold=0.4. On the 25-sample rendered pack it still beat raw Tesseract on overall CER (0.248654 vs 0.689214) and kept a large structure advantage (table_cell_f1 0.955556 vs 0.0). The weak spots are still visible, which is exactly why this should be treated as an engineering baseline rather than a claim candidate: form CER regressed slightly and aggregate reading-order accuracy on that pack is still lower than Tesseract.

Table Cell Refinement Composite

table_cell_refine_composite uses one engine for document/table structure and a second engine for crop OCR over detected table cells:

ocrtech create-model-card \
  --out outputs/table-cell-refine-empty-v1/model-card.json \
  --model-id ocrtech-surya-tesseract-table-cell-refine-empty-v1 \
  --backend table_cell_refine_composite \
  --base-model surya+tesseract \
  --backend-kwarg structure_engine=surya \
  --backend-kwarg crop_engine=tesseract \
  --backend-kwarg crop_padding=4 \
  --backend-kwarg replacement_policy=empty

Policies:

  • replacement_policy=empty: only OCR empty table cells. This is the safe default.
  • replacement_policy=always: reread every detected table cell. This is experimental and can damage already-correct Surya cells.
  • deduplicate_tables=true: drop overlapping duplicate tables after refinement. The parser also avoids re-inferring a table from text lines when an engine already supplied an overlapping structured table.

Current hard15 evidence says the idea is not enough:

  • unsafe always policy tied Surya on text CER but regressed table-cell F1 from 0.979167 to 0.593750
  • safe empty policy tied Surya exactly on hard15 CER, table-cell F1, and reading order
  • safe empty plus table de-duplication also tied Surya exactly on hard15 CER, table-cell F1, and reading order
  • hard15 validation status for the safe policies is not_validated

So this backend is useful infrastructure for future missing-cell cases, not a claim candidate yet.

Figure Caption Refinement Composite

figure_caption_refine_composite keeps a primary document engine for layout and OCR, then uses a second engine only to replace nearby figure-caption lines when the replacement overlaps spatially and has a better caption-prefix score:

ocrtech create-model-card \
  --out outputs/figure-caption-refine-v1/model-card.json \
  --model-id ocrtech-surya-tesseract-figure-caption-refine-v1 \
  --backend figure_caption_refine_composite \
  --base-model surya+tesseract \
  --backend-kwarg primary_engine=surya \
  --backend-kwarg caption_engine=tesseract \
  --backend-kwarg min_caption_iou=0.5 \
  --backend-kwarg min_score_delta=1

The replacement preserves the primary engine's bbox and layout ordering. Only the caption text/confidence changes. This avoids the skew failure where a secondary OCR bbox overlaps the figure marker and prevents caption attachment.

Current hard15 evidence:

  • figure-caption CER improved over Surya from 0.092323 to 0.052478
  • figure detection F1 stayed 1.000000
  • table-cell F1 stayed 0.979167
  • reading-order pair accuracy stayed 0.520000
  • hard15 validation status is still not_validated because broad CER/table gates do not pass

An opt-in correct_structural_labels=true branch adds edit-distance guarded correction for common Nepali document-role labels such as वार्षिक प्रतिवेदन, चित्र, and निष्कर्ष. On hard15 this stronger branch improved:

  • overall hard_eval CER over Surya from 0.236839 to 0.204610
  • figure CER from 0.091927 to 0.022866
  • figure-caption CER from 0.092323 to 0.008746

Validation still reports not_validated because table CER and table-cell F1 tie Surya, but the overall CER gate and both figure gates pass.

A stronger auditable variant keeps the same backend but enables:

  • correct_structural_labels=true, now covering वार्षिक प्रतिवेदन, दैनिक हिसाब, चित्र, and निष्कर्ष
  • table_name_lexicon_path=configs/nepali-table-name-lexicon-v1.txt
  • table_name_max_distance=2

The table-name lexicon correction is intentionally narrow: it only applies to cells under a header close to नाम, and only when there is a unique nearest lexicon entry within the configured edit-distance threshold. The current lexicon is tiny and curated, so paper claims must describe this as lexicon-assisted structured extraction, not a general language model.

The public-table variant also enables:

  • normalize_table_grids=true

This removes empty columns, separator-only columns such as —, and exact duplicate adjacent columns from predicted tables. The rule is runtime-only and reference-free. It improved the 5-page public OmniDocBench smoke over raw Surya on table metrics, but the public validation still remains not_validated because the sample count and confidence gates are not met.

The later normalize-rows-v3 public-table model card keeps the same option and adds sparse bullet row-fragment merging inside the grid normalizer. This is also runtime-only and reference-free. On the same 5-page public OmniDocBench smoke it raises table metrics further (table_cell_f1 0.564335, table_teds_structure 0.942836, table_teds_text 0.675633) and changes the omnidocbench-00208... fragmented table from 16x5 to 3x5. It is not a validated model: text CER is slightly worse than Surya on that smoke, and the public validation gates still fail on sample count and bootstrap CI.

On the larger public OmniDocBench table40 slice, the same model has higher mean table metrics than Surya but does not validate:

  • CER 0.713326 versus Surya 0.713960
  • table-cell F1 0.323122 versus Surya 0.280062, but 7 wins / 9 losses / 24 ties and CI low -0.019152
  • table TEDS structure 0.520959 versus Surya 0.503603, but CI low -0.038381
  • table TEDS text 0.361666 versus Surya 0.358057, below the configured margin
  • cell exactness 3440/10314 versus Surya 3421/10314

This points to the next model requirement: better table segmentation and large/wide-table handling. The current grid postprocessors are useful but too local to support a public SOTA claim.

The normalize-width-guard-v4 model card adds one guard to the same runtime postprocessor: tables wider than 12 predicted columns skip grid normalization. This directly addresses public table40 failures where v3 merged 15 sparse rows or dropped 26 columns in wide tables. On full public table40, v4 improves over v3 and passes two configured public table gates against Surya:

  • CER 0.707864 versus Surya 0.713960
  • table-cell F1 0.339319 versus Surya 0.280062, CI low 0.001433
  • table TEDS structure 0.545375 versus Surya 0.503603, CI low 0.004790
  • table TEDS text 0.386013 versus Surya 0.358057, but CI low -0.000220
  • cell exactness 3453/10314 versus Surya 3421/10314

This is a better public table parser candidate than v3, but still not a SOTA engine: the text-sensitive table gate and broad public document gate remain open.

On hard15 v3, this model reaches:

  • overall CER 0.010671 versus Surya 0.236839
  • table-slice CER 0.000000 versus Surya 0.363636
  • figure-caption CER 0.008746 versus Surya 0.092323
  • table-cell F1 1.000000 versus Surya 0.979167
  • table exact cells 96/96 versus Surya 94/96

Configured validation is still not_validated because the table-cell F1 bootstrap CI low is 0.000000 on the tiny 8-page table slice. This is a sample-size limitation of the current gate, not permission to claim table SOTA. Build a larger independent table pack before making that claim.

The table-name lexicon matcher now also supports suffix-preserving generated IDs such as करिण-302 -> किरण-302. The correction still only fires in columns whose header is close to नाम, and still requires a unique nearest base-name match within the configured edit distance.

On the larger 60-sample table-ledger pack, this same model passes the targeted Surya table validation:

  • table-cell F1: candidate 1.000000, Surya 0.979167, CI low 0.012500
  • table exact match: candidate 1.000000, Surya 0.750000, CI low 0.150000
  • table structure similarity: candidate 1.000000, Surya 1.000000, non-regression guard passes
  • cell exactness: candidate 720/720, Surya 705/720

This is valid evidence for lexicon-assisted table-ledger cell-content extraction with grid-structure non-regression. It is not a general OCR SOTA result, and it does not prove improvement on complex merged-cell tables.

On the first complex-table-ledger run, the same candidate improved cell-content F1 over Surya (0.928333 vs 0.915000) after extending name correction below grouped headers, but it did not improve merged-header structure.

The later span-inference parser branch adds conservative repeated-header colspan inference. On the same generated complex-table pack, it reaches:

  • table-cell F1 0.995000 versus Surya 0.915000
  • table exact match 0.925000 versus Surya 0.000000
  • span-aware table structure similarity 1.000000 versus Surya 0.875000
  • TEDS-style table tree structure similarity 1.000000 versus Surya 0.904762
  • TEDS-style text-sensitive table tree similarity 0.996250 versus Surya 0.891667
  • cell exactness 597/600 versus Surya 549/600

This is a parser/postprocessing improvement for a generated repeated-header pattern. It is not a trained merged-cell model and does not support a broad complex-table SOTA claim without real scanned tables and varied span structures.

The public OmniDocBench table40 width-guard model card is:

  • outputs/figure-caption-table-lexicon-normalize-width-guard-v4/model-card.json
  • model id: ocrtech-surya-tesseract-figure-table-lexicon-normalize-width-guard-v4
  • audit: outputs/audit-figure-caption-table-lexicon-normalize-width-guard-v4-model/model-audit.json

This candidate keeps the normalized table-grid postprocessors but skips grid normalization when predicted table width is greater than 12. On the public OmniDocBench table40 combined run, it improves over Surya on table-cell F1 (0.339319 vs 0.280062) and structure-only TEDS (0.545375 vs 0.503603), but the run remains not_validated because the public document CER gate is underpowered and table TEDS text is borderline in the separate-run comparison.

For postprocessor-only evidence, use outputs/experiment-public-omnidocbench-table40-derived-postprocess-v1/benchmark/report.json. That derived benchmark applies the table postprocessors to the exact same Surya document.json outputs, so CER stays identical while table metrics isolate the postprocessor contribution. It passes the configured table-cell F1, structure-only TEDS, and text-sensitive TEDS gates on 40 pages, but it is a validation lens rather than a deployable model card.

Audit

ocrtech audit-model runs/recognizer/package/model-card.json \
  --out outputs/model-audit

The audit verifies:

  • required artifact paths exist
  • recorded SHA-256 values match for file artifacts
  • backend is supported
  • the card shape is valid

audit-model does not by itself prove claim readiness. For quality_select_composite, quality_ranked_ensemble, and script_select_composite, the calibration provenance must also be separate from the final claim eval manifest.

For recognizer branch admission, attach the hard-gate validation report when creating or packaging a model card:

ocrtech create-model-card \
  --out runs/recognizer/package/model-card.json \
  --model-id ocrtech-next-recognizer \
  --backend paddleocr \
  --artifact runs/recognizer/export/inference/inference.pdiparams \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json

Claim-grade preflight can then require the admission gate with --require-model-admission. Admission reports must include quantitative passing gate evidence; a bare claim_status=validated report is rejected.

Parse With Candidate

ocrtech parse-doc page.png \
  --engine candidate \
  --model-config runs/recognizer/package/model-card.json \
  --fallback-model-config runs/vlm-fallback/package/model-card.json \
  --out outputs/page

When a fallback engine or fallback model card is configured, the parser does not silently swap engines. It records metadata.fallback in document.json with the primary engine, fallback engine/model config, average primary confidence, threshold, trigger reason, and outcome. This is required for paper evidence: a selective VLM/document fallback must report when it actually fired and whether it succeeded.

Benchmark With Candidate

ocrtech run-experiment \
  --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --references-dir data/eval-pack/references \
  --candidate-model-config runs/recognizer/package/model-card.json \
  --fallback-model-config runs/vlm-fallback/package/model-card.json \
  --low-confidence-threshold 0.80 \
  --baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
  --validation-config validation.json \
  --train-manifest data/splits/train.jsonl \
  --out outputs/claim-experiment \
  --require-validated

If --candidate-model-config is omitted, ours uses the local auto engine for smoke tests. That is not valid SOTA evidence.

Benchmark and experiment fallback flags apply only to the candidate/ours run. They are meant to measure the deployed hybrid engine, not to contaminate baseline rows. experiment.json records fallback_engine, fallback_model_config, fallback_model_config_sha256, and low_confidence_threshold so claim reviewers can tell which routing policy was actually evaluated.