File size: 18,755 Bytes
f9bc0c1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
# ocrtech CLI reference

Moved verbatim from the README on 2026-09-24. Host addresses in the remote examples are
historical; use the `k2` SSH alias. Install the package first (see the README).

## CLI

```bash
ocrtech prepare-data labels.csv --out data/manifests --image-field image_path --text-field text
ocrtech prepare-data labels.csv --out data/manifests --image-field image_path --text-field text --skip-invalid
ocrtech prepare-data gauravgiri/nepali-ocr-dataset --hf --split train --limit 100 --out data/manifests
ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines --hf --split train --limit 100 --out data/manifests
ocrtech prepare-data nvidia/OCR-Synthetic-Multilingual-v1 --hf --hf-subset en --split test --limit 1000 --out data/manifests-english-test
ocrtech merge-manifests data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl --out data/manifests/nepali-mixed-all.jsonl
ocrtech audit-manifest data/manifests/labels-train.jsonl --require-slice nepali --require-slice english
ocrtech split-manifest data/manifests/labels-train.jsonl --out data/splits --eval-ratio 0.15
ocrtech rebalance-manifest data/splits/train.jsonl --out data/splits/train-rebalanced.jsonl --slice english --slice nepali
ocrtech audit-recognizer-corpus --train-manifest data/splits/train-rebalanced.jsonl --eval-manifest data/splits/eval.jsonl --out outputs/recognizer-corpus-audit
ocrtech build-recognizer-corpus data/manifests/english-lines.jsonl data/manifests/nepali-lines.jsonl data/manifests/mixed-crops.jsonl --out data/splits/recognizer-train-balanced.jsonl --target-latin-only 200 --target-devanagari-only 200 --target-mixed 100 --target-real 100 --target-synthetic 100
ocrtech materialize-references data/splits/eval.jsonl --out data/references --rewritten-manifest data/splits/eval-with-refs.jsonl
ocrtech create-eval-pack --out data/eval-pack --input-format image --degradation clean --degradation scan
ocrtech create-eval-pack --out data/eval-pack-table-figure-dev-v1 --input-format image --template table-ledger --template figure-caption --count-per-template 8 --variant-offset 100 --degradation clean --degradation scan --degradation blur --degradation low_contrast --degradation skew
ocrtech audit-eval-pack data/eval-pack/manifests/ocrtech-eval-pack.jsonl --require-claim-ready
ocrtech train-recognizer --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --out runs/recognizer
ocrtech train-recognizer --backend hf-vision-encoder-decoder --base-model microsoft/trocr-small-printed --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --out runs/hf-recognizer
ocrtech bundle-recognizer-training --recipe-dir runs/recognizer --train-manifest data/splits/train-rebalanced.jsonl --eval-manifest data/splits/eval.jsonl --out runs/recognizer-training-bundle --require-corpus-audit
ocrtech export-recognizer --training-config runs/recognizer/paddle_recognizer.yml --checkpoint runs/recognizer/checkpoints/best_accuracy --out runs/recognizer/export
ocrtech package-recognizer --inference-dir runs/recognizer/export/inference --dictionary runs/recognizer/nep_eng_dict.txt --out runs/recognizer/package --model-id ocrtech-nep-eng-rec-v1 --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --training-summary runs/recognizer/recognizer-training-summary.json --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
ocrtech package-hf-recognizer --model-dir runs/hf-recognizer/final-model --out runs/hf-recognizer/package --model-id ocrtech-hf-nep-eng-rec-v1 --base-model microsoft/trocr-small-printed --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --training-summary runs/hf-recognizer/recognizer-training-summary.json --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
ocrtech train-corrector --clean-manifest data/splits/train.jsonl --out runs/corrector
ocrtech train-corrector --hf-dataset cfilt/RoundTripOCR-nepali --hf-split train --limit 1000 --out runs/corrector-cfilt
ocrtech audit-model runs/recognizer/package/model-card.json
ocrtech audit-remote-host --host 100.97.238.103 --user k2 --password-env OCRTECH_REMOTE_PASSWORD --workdir '~/ocr-tech' --require-path '~/ocr-tech' --min-free-gb 20 --require-paddle-training --out outputs/remote-audit
ocrtech bootstrap-remote-host --host 100.97.238.103 --user k2 --password-env OCRTECH_REMOTE_PASSWORD --workdir '~/ocr-tech' --out outputs/remote-bootstrap
ocrtech sync-remote-workspace --host 100.97.238.103 --user k2 --password-env OCRTECH_REMOTE_PASSWORD --local-root . --remote-workdir '~/ocr-tech' --exclude .venv --exclude outputs --exclude runs --out outputs/remote-sync
ocrtech preflight-claim --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl --references-dir data/eval-pack/references --baselines candidate,tesseract,stock-paddle --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json --train-manifest data/splits/train.jsonl --out outputs/preflight
ocrtech parse-doc examples/sample-page.txt --out outputs/sample --engine sidecar
ocrtech parse-doc page.png --out outputs/page --engine candidate --model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json
ocrtech benchmark examples/sample-page.txt --out outputs/bench --baselines ours --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json
ocrtech summarize-benchmark --benchmark-report outputs/bench/report.json --out outputs/bench-summary --candidate ours --baseline tesseract --metric cer
ocrtech benchmark-error-analysis --benchmark-report outputs/bench/report.json --out outputs/bench-error-analysis --system surya --baseline tesseract --metric cer
ocrtech export-failure-manifest --source-manifest data/splits/eval.jsonl --top-failures outputs/bench-error-analysis/top-failures.json --out data/splits/eval-hard.jsonl --split hard_eval
ocrtech recognizer-admission-decision --validation-report outputs/validation-next-recognizer-hard50/validation.json --out outputs/recognizer-admission-next
ocrtech paper-benchmark-table --summary outputs/paper-claimpack-summary-tesseract/summary.json --summary outputs/paper-claimpack-summary-surya/summary.json --summary outputs/paper-claimpack-summary-stock-paddle/summary.json --out outputs/paper-claimpack-table --system candidate --system surya --system tesseract --system stock-paddle --label 'candidate=Candidate' --label 'surya=Surya' --label 'tesseract=Tesseract' --label 'stock-paddle=Stock Paddle' --source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' --source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' --source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' --source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' --metric cer --metric table_cell_f1 --metric reading_order_pair_accuracy --metric latency_seconds
ocrtech calibrate-quality-router --experiment-report outputs/calibration-experiment/experiment.json --out outputs/quality-router-calibration --model-id ocrtech-tesseract-quality-router --primary-engine tesseract --secondary-engine tesseract --primary-engine-kwarg 'language=nep+eng' --primary-engine-kwarg 'psm=6' --secondary-engine-kwarg 'language=nep+eng' --secondary-engine-kwarg 'psm=13'
ocrtech calibrate-script-router --experiment-report outputs/script-calibration/experiment.json --out outputs/script-router-calibration --model-id ocrtech-tesseract-surya-script-router --primary-engine tesseract --secondary-engine surya --primary-engine-kwarg 'language=nep+eng' --primary-engine-kwarg 'psm=6' --secondary-engine-kwarg 'mode=auto'
ocrtech calibrate-tesseract-psm-ensemble --primary-experiment-report outputs/psm13-calibration/experiment.json --variant-experiment psm7=outputs/psm7-calibration/experiment.json --variant-experiment psm11=outputs/psm11-calibration/experiment.json --variant-experiment psm13=outputs/psm13-calibration/experiment.json --out outputs/psm-ensemble-calibration --model-id ocrtech-tesseract-psm-ensemble
ocrtech run-experiment --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl --out outputs/experiment --baselines candidate,tesseract,stock-paddle --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json
ocrtech review-claim --experiment outputs/internal-claim/experiment.json --experiment outputs/ocrbench-v2/experiment.json --config claim-review.json --model-config runs/recognizer/package/model-card.json --out outputs/claim-review
ocrtech validate-claim --benchmark-report outputs/bench/report.json --config validation.json --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json --out outputs/validation
```

`parse-doc` writes:

- `document.json`
- `document.md`
- `tables/*.csv`
- `tables/*.html`
- `figures/*.png`
- `figures/metadata.json`
- `quality.json`

Figure exports are explicit. If an engine supplies `figure.image_path`,
`parse-doc` copies that file into `figures/`. If an engine supplies only a
figure bbox and the source is a regular image or page-bundle directory,
`parse-doc` crops the region into `figures/{figure_id}.png` when Pillow is
available. For bundles, it uses `figure.page_index` to select the page image. If
the artifact cannot be created, `figures/metadata.json` records
`metadata.export_warnings` instead of silently dropping the figure image.

When `--fallback-engine` or `--fallback-model-config` is configured,
`document.json` records `metadata.fallback` with the primary engine, fallback
engine/model card, confidence threshold, trigger reason, and success/error
outcome. Benchmarks surface this as diagnostic metrics such as
`fallback_triggered`, `fallback_succeeded`, and `fallback_failed`.
The same fallback flags are available on `benchmark` and `run-experiment` for
candidate/ours runs, and experiment metadata records the fallback model-card
path, SHA-256, and confidence threshold.

## Engines

`auto` tries deterministic sidecar OCR first, then PaddleOCR, then Tesseract. Optional engines fail loudly when binaries or packages are missing.

Sidecar inputs are useful for tests and staged pipelines. For `page.png`, provide one of:

- `page.png.ocr.json`
- `page.ocr.json`
- `page.png.txt`
- `page.txt`

## Data

Use `prepare-data` to verify actual fields and image paths before training. By default it fails on the first bad row. Use `--skip-invalid` when the upstream dataset is messy and you want a usable manifest plus an explicit reject log instead of a silent partial corpus.

Use `merge-manifests` to assemble multi-source corpora without ad hoc `cat` pipelines, then `audit-manifest` before training and `split-manifest` to create deterministic leakage-safe train/eval manifests. The split groups exact sample signatures before assignment so duplicate image+text examples cannot land on both sides.

Known Hugging Face adapters now stamp verified slice metadata automatically:

- `gauravgiri/nepali-ocr-dataset`: `nepali`, `real`
- `prashant0919/nepali-synthetic-ocr-lines`: `nepali`, `synthetic`
- `nvidia/OCR-Synthetic-Multilingual-v1 --hf-subset en`: `english`, `synthetic`, `line_crop`, `en`

Use `rebalance-manifest` only on the training split after `split-manifest`. Rebalancing before the split would turn deliberate duplication into leakage.

Use `audit-recognizer-corpus` before launching a recognizer training job. It
checks the actual train/eval manifests for sample counts, Latin-script coverage,
Devanagari coverage, mixed-script examples, real/synthetic training coverage,
and train/eval leakage. This guardrail was added after the failed PaddleOCR
mixed v4 branch: a new GPU run should not start unless English preservation and
Nepali coverage are explicit in the corpus.

Use `build-recognizer-corpus` to assemble and upsample the training manifest by
actual text buckets, not by wishful dataset labels. It tracks `latin_only`,
`devanagari_only`, `mixed_devanagari_latin`, `real`, and `synthetic` coverage,
then writes a build JSON/Markdown summary next to the output manifest. It does
not invent missing categories: if a required bucket is absent from the input
manifests, the command fails so you know to add real source data before using
GPU time.

`train-recognizer --backend paddleocr` writes
`recognizer-training-summary.json` alongside the generated PaddleOCR config.
Keep that file with any remote/GPU training bundle and pass it to
`package-recognizer --training-summary`; it records manifest hashes, slice
counts, character coverage, label/config hashes, and launch hyperparameters.
`bundle-recognizer-training` records the bundled summary path and SHA-256 under
`training_summary`, and packaged recognizer model cards record the same
provenance when supplied. For real remote/GPU runs, use
`bundle-recognizer-training --require-corpus-audit`; it runs the recognizer
corpus audit and refuses to create the bundle if the corpus does not meet the
configured English, Devanagari, mixed-script, real/synthetic, and leakage gates.
The gate also checks isolated Latin-only and Devanagari-only samples; a corpus
made entirely of mixed-script rows is not enough to protect English preservation
or Nepali-only recognition.

Planned source datasets:

- `gauravgiri/nepali-ocr-dataset`
- `prashant0919/nepali-synthetic-ocr-lines`
- `nvidia/OCR-Synthetic-Multilingual-v1`
- `cfilt/RoundTripOCR-nepali`

The verified Hugging Face adapters are intentionally dataset-specific where the upstream schema is non-standard:

- `gauravgiri/nepali-ocr-dataset`: generic `image` + `text`
- `prashant0919/nepali-synthetic-ocr-lines`: `metadata.jsonl` plus per-line image files, downloaded lazily for the requested rows
- `cfilt/RoundTripOCR-nepali`: correction pairs via `ocr` and `correct`
- `nvidia/OCR-Synthetic-Multilingual-v1`: HDF5 shards with `annotations.line_bboxes`; use `--hf-subset en` for English crops

Some upstream corpora are dirty. For example, the current `gauravgiri/nepali-ocr-dataset` sample prefix includes null `text` rows, so use `--skip-invalid` when you want a prepared manifest plus an explicit reject log instead of a hard stop.

## Baselines

The benchmark command supports:

- `ours`
- `tesseract`
- `stock-paddle`
- `glm-ocr`
- `paddleocr-vl`

`glm-ocr` and `paddleocr-vl` are external command hooks:

```bash
export OCRTECH_GLM_OCR_CMD='python scripts/run_glm_ocr.py --input {input} --out {out}'
export OCRTECH_PADDLEOCR_VL_CMD='python scripts/run_paddleocr_vl.py --input {input} --out {out}'
```

The command must include both `{input}` and `{out}` placeholders. It may print Markdown to stdout or write `{out}/document.md`; `ocrtech` now normalizes that Markdown into its own `document.json` schema so external VLM baselines can still be scored for text, table cells, and reading order.

The bundled wrappers are thin adapters, not magic:

- `scripts/run_glm_ocr.py` uses the official `transformers` path from the `zai-org/GLM-OCR` model card.
- `scripts/run_paddleocr_vl.py` uses the official `PaddleOCRVL(...).predict(...)` path from the `PaddlePaddle/PaddleOCR-VL-1.6` model card.
- GLM-OCR still expects a git `transformers` build upstream. On `k2`, the released `transformers 4.57.6` package was not enough to instantiate its processor classes; a dedicated Python 3.11 env with git `transformers` and `torchvision` was required before the wrapper would run.
- PaddleOCR-VL still needs the OCR extras that the official docs call out, for example `pip install "paddlex[ocr]==3.7.1"`.

## Validation Before SOTA Claims

Use `validate-claim` before making any quality claim. The default validator is intentionally strict and will fail unless the benchmark report has held-out sample coverage, required slices, leakage checks, paired candidate-vs-baseline gates, and per-gate evidence such as paired wins and optional significance thresholds.

Use `recognizer-admission-decision` after a recognizer admission validation run.
It writes a small JSON/Markdown decision artifact: `admit` only for
`claim_status=validated` with `passed=true`, otherwise `reject` with failed
gate metrics and next actions for the next model branch.

Use `create-eval-pack` for internal mixed Nepali/English document tests and `audit-eval-pack` to check whether the pack is claim-ready. The internal pack now includes a real `multi_page` slice in addition to the single-page paragraph, table, form, and figure templates. Text packs are for pipeline smoke tests only; image packs are required for OCR claims.

Use `preflight-claim` before `run-experiment` on any claim-grade run. It fails early on missing references, invalid model cards, missing baseline dependencies, non-claim-ready eval packs, bad validation configs, and train/eval leakage.

For routed or thresholded candidates such as `quality_select_composite` and `script_select_composite`, claim-grade runs also require explicit calibration provenance in the model card. If the candidate was calibrated on the same eval manifest you are about to benchmark, `preflight-claim` now fails on purpose.

For those routed candidates, claim-grade runs also need the raw component baselines in the same benchmark. If a `script_select_composite` candidate is built from Tesseract and Surya, the claim run must include `candidate,tesseract,surya`; otherwise the repo now fails preflight because distinctness cannot be checked honestly.

Use `run-experiment` to benchmark every manifest input, write reproducibility metadata, and run claim validation in one output directory.

Use `review-claim` after the internal pack and public benchmark suites have all been run. It audits the portfolio-level evidence bundle so one green experiment does not get misrepresented as a broad SOTA claim.

Use a candidate model card for any real benchmark of `ours`; see [docs/MODELS.md](MODELS.md).
Use `filter-manifest` when a baseline is too expensive to run on the full pack and you need a reproducible exploratory subset instead of hand-editing a JSONL.

See [docs/PREFLIGHT.md](PREFLIGHT.md), [docs/CLAIM_REVIEW.md](CLAIM_REVIEW.md), [docs/EVAL_PACK.md](EVAL_PACK.md), [docs/MODELS.md](MODELS.md), [docs/EXPERIMENTS.md](EXPERIMENTS.md), and [docs/VALIDATION.md](VALIDATION.md).

## Remote Training

Use `docs/REMOTE_TRAINING.md` for the current `k2` host notes, `audit-remote-host`, and the safe SSH workflow. Do not commit SSH passwords or machine-specific secrets.