diff --git a/ocr-tech-mps-integration/.gitignore b/ocr-tech-mps-integration/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..7675c338d8ea297adef8136c7bb5ad77153fd03d --- /dev/null +++ b/ocr-tech-mps-integration/.gitignore @@ -0,0 +1,19 @@ +.venv/ +__pycache__/ +*.py[cod] +.pytest_cache/ +.ruff_cache/ +.mypy_cache/ + +data/* +!data/dictionaries/ +!data/dictionaries/*.txt +outputs/ +runs/ +*.egg-info/ + +.DS_Store + +spaces/limbu-ocr/models/ +incoming/ +logs/ diff --git a/ocr-tech-mps-integration/.ignore b/ocr-tech-mps-integration/.ignore new file mode 100644 index 0000000000000000000000000000000000000000..802a8479566fbec8d4078d25e2ae507bd9a55054 --- /dev/null +++ b/ocr-tech-mps-integration/.ignore @@ -0,0 +1,9 @@ +.venv/ +__pycache__/ +*.py[cod] +.pytest_cache/ + +data/ +outputs/ +runs/ +*.egg-info/ diff --git a/ocr-tech-mps-integration/CLAUDE.md b/ocr-tech-mps-integration/CLAUDE.md new file mode 100644 index 0000000000000000000000000000000000000000..51e85d9878b44439f46d492105fee8c23c3ac09a --- /dev/null +++ b/ocr-tech-mps-integration/CLAUDE.md @@ -0,0 +1,38 @@ +# ocr-tech — agent instructions + +## Drive documents + +- `RESEARCH.md` is the research state and drive document. Read it first every + session; keep it current when results land. +- End goal: OCR covering the whole Nepal language pack, **Limbu first, Newar + (Nepal Bhasa) second** (steer 2026-07-04). + +## Model delegation policy (steer 2026-07-04) + +- **Fable (lead session model)**: orchestration, the hardest reasoning, design + decisions, and final verification of all delegated work. Best model available — + do not delegate judgment calls. +- **Codex**: treat as a peer, not a subordinate. It has good independent ways of + figuring things out. Hand it substantial, well-scoped build/diagnosis/research + tasks (new tools, refactors, root-cause hunts) with verified file paths and + acceptance criteria; always verify its output independently (run its tests, + spot-check its data). +- **Opus**: near-par with the lead model but slightly weaker; usable for hard + tasks and second opinions when a peer-level check is wanted. +- **Sonnet**: tasks that require no big thought — mechanical edits, bulk file + wrangling, running known command recipes, simple searches and summaries. + +## Working rules + +- Never assume file paths, schemas, or endpoint behavior — verify on disk / live + before coding against them (this repo's history has multiple wrong-assumption + scars; see RESEARCH.md "Mistakes To Avoid"). +- Real CER comes from `tools/eval_metrics.py` (micro CER, codepoint + grapheme). + Never report Paddle `norm_edit_dis` as CER. +- Silver labels are machine-converted; they are training/diagnostic substrate, + not claim-grade ground truth. Claim-grade numbers need frozen, human-reviewed + eval packs. +- Long downloads/runs: launch detached (`nohup`), not under tool timeouts; watch + with polling loops. `ocrtech audit-gorkhapatra-language-pages` has no resume. +- Python runtime for Paddle work is 3.11 (`uv run --python 3.11`); silver tooling + needs `--with pymupdf --with npttf2utf`. diff --git a/ocr-tech-mps-integration/README.md b/ocr-tech-mps-integration/README.md new file mode 100644 index 0000000000000000000000000000000000000000..d793ffe05dbdc40e1735d120b846c2d9b11acd67 --- /dev/null +++ b/ocr-tech-mps-integration/README.md @@ -0,0 +1,371 @@ +# ocr-tech + +Hybrid Nepali + English OCR and document-structure extraction toolkit. + +This repository is a runnable foundation for the document-understanding plan: + +- local OCR/structure pipeline first +- optional PaddleOCR/Tesseract/remote model adapters +- stable CLI commands for data prep, training recipes, parsing, and benchmarks +- explicit output schema for Markdown, JSON, tables, and figures + +It does not claim that the fine-tuned recognizer is already trained. Training commands generate checked manifests, PaddleOCR label files, dictionaries, and recipe configs so long jobs can be run on a suitable host. + +The active research steering checklist is tracked in +[docs/ACTIVE_RESEARCH_CHECKLIST.md](docs/ACTIVE_RESEARCH_CHECKLIST.md). It +records the cdjk remote synthesis leads, Gorkhapatra validation path, and paper +claim gates. + +## Install + +```bash +python3 -m venv .venv +. .venv/bin/activate +python -m pip install -e '.[dev]' +``` + +Optional integrations: + +```bash +python -m pip install -e '.[eval]' +python -m pip install -e '.[datasets,engines]' +python -m pip install -e '.[hf-recognizer]' +``` + +The `eval` extra installs Pillow and PyMuPDF for image/PDF evaluation artifacts, +including Gorkhapatra epaper page rendering. + +The `hf-recognizer` extra is the real dependency set for TrOCR-style training and inference here. It includes `torch`, `transformers`, `accelerate`, `sentencepiece`, `protobuf`, and `pillow`. + +## CLI + +```bash +ocrtech prepare-data labels.csv --out data/manifests --image-field image_path --text-field text +ocrtech prepare-data labels.csv --out data/manifests --image-field image_path --text-field text --skip-invalid +ocrtech prepare-data gauravgiri/nepali-ocr-dataset --hf --split train --limit 100 --out data/manifests +ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines --hf --split train --limit 100 --out data/manifests +ocrtech prepare-data nvidia/OCR-Synthetic-Multilingual-v1 --hf --hf-subset en --split test --limit 1000 --out data/manifests-english-test +ocrtech merge-manifests data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl --out data/manifests/nepali-mixed-all.jsonl +ocrtech audit-manifest data/manifests/labels-train.jsonl --require-slice nepali --require-slice english +ocrtech split-manifest data/manifests/labels-train.jsonl --out data/splits --eval-ratio 0.15 +ocrtech rebalance-manifest data/splits/train.jsonl --out data/splits/train-rebalanced.jsonl --slice english --slice nepali +ocrtech audit-recognizer-corpus --train-manifest data/splits/train-rebalanced.jsonl --eval-manifest data/splits/eval.jsonl --out outputs/recognizer-corpus-audit +ocrtech build-recognizer-corpus data/manifests/english-lines.jsonl data/manifests/nepali-lines.jsonl data/manifests/mixed-crops.jsonl --out data/splits/recognizer-train-balanced.jsonl --target-latin-only 200 --target-devanagari-only 200 --target-mixed 100 --target-real 100 --target-synthetic 100 +ocrtech materialize-references data/splits/eval.jsonl --out data/references --rewritten-manifest data/splits/eval-with-refs.jsonl +ocrtech create-eval-pack --out data/eval-pack --input-format image --degradation clean --degradation scan +ocrtech create-eval-pack --out data/eval-pack-table-figure-dev-v1 --input-format image --template table-ledger --template figure-caption --count-per-template 8 --variant-offset 100 --degradation clean --degradation scan --degradation blur --degradation low_contrast --degradation skew +ocrtech audit-eval-pack data/eval-pack/manifests/ocrtech-eval-pack.jsonl --require-claim-ready +ocrtech train-recognizer --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --out runs/recognizer +ocrtech train-recognizer --backend hf-vision-encoder-decoder --base-model microsoft/trocr-small-printed --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --out runs/hf-recognizer +ocrtech bundle-recognizer-training --recipe-dir runs/recognizer --train-manifest data/splits/train-rebalanced.jsonl --eval-manifest data/splits/eval.jsonl --out runs/recognizer-training-bundle --require-corpus-audit +ocrtech export-recognizer --training-config runs/recognizer/paddle_recognizer.yml --checkpoint runs/recognizer/checkpoints/best_accuracy --out runs/recognizer/export +ocrtech package-recognizer --inference-dir runs/recognizer/export/inference --dictionary runs/recognizer/nep_eng_dict.txt --out runs/recognizer/package --model-id ocrtech-nep-eng-rec-v1 --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --training-summary runs/recognizer/recognizer-training-summary.json --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json +ocrtech package-hf-recognizer --model-dir runs/hf-recognizer/final-model --out runs/hf-recognizer/package --model-id ocrtech-hf-nep-eng-rec-v1 --base-model microsoft/trocr-small-printed --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --training-summary runs/hf-recognizer/recognizer-training-summary.json --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json +ocrtech train-corrector --clean-manifest data/splits/train.jsonl --out runs/corrector +ocrtech train-corrector --hf-dataset cfilt/RoundTripOCR-nepali --hf-split train --limit 1000 --out runs/corrector-cfilt +ocrtech audit-model runs/recognizer/package/model-card.json +ocrtech audit-remote-host --host 100.97.238.103 --user k2 --password-env OCRTECH_REMOTE_PASSWORD --workdir '~/ocr-tech' --require-path '~/ocr-tech' --min-free-gb 20 --require-paddle-training --out outputs/remote-audit +ocrtech bootstrap-remote-host --host 100.97.238.103 --user k2 --password-env OCRTECH_REMOTE_PASSWORD --workdir '~/ocr-tech' --out outputs/remote-bootstrap +ocrtech sync-remote-workspace --host 100.97.238.103 --user k2 --password-env OCRTECH_REMOTE_PASSWORD --local-root . --remote-workdir '~/ocr-tech' --exclude .venv --exclude outputs --exclude runs --out outputs/remote-sync +ocrtech preflight-claim --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl --references-dir data/eval-pack/references --baselines candidate,tesseract,stock-paddle --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json --train-manifest data/splits/train.jsonl --out outputs/preflight +ocrtech parse-doc examples/sample-page.txt --out outputs/sample --engine sidecar +ocrtech parse-doc page.png --out outputs/page --engine candidate --model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json +ocrtech benchmark examples/sample-page.txt --out outputs/bench --baselines ours --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json +ocrtech summarize-benchmark --benchmark-report outputs/bench/report.json --out outputs/bench-summary --candidate ours --baseline tesseract --metric cer +ocrtech benchmark-error-analysis --benchmark-report outputs/bench/report.json --out outputs/bench-error-analysis --system surya --baseline tesseract --metric cer +ocrtech export-failure-manifest --source-manifest data/splits/eval.jsonl --top-failures outputs/bench-error-analysis/top-failures.json --out data/splits/eval-hard.jsonl --split hard_eval +ocrtech recognizer-admission-decision --validation-report outputs/validation-next-recognizer-hard50/validation.json --out outputs/recognizer-admission-next +ocrtech paper-benchmark-table --summary outputs/paper-claimpack-summary-tesseract/summary.json --summary outputs/paper-claimpack-summary-surya/summary.json --summary outputs/paper-claimpack-summary-stock-paddle/summary.json --out outputs/paper-claimpack-table --system candidate --system surya --system tesseract --system stock-paddle --label 'candidate=Candidate' --label 'surya=Surya' --label 'tesseract=Tesseract' --label 'stock-paddle=Stock Paddle' --source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' --source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' --source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' --source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' --metric cer --metric table_cell_f1 --metric reading_order_pair_accuracy --metric latency_seconds +ocrtech calibrate-quality-router --experiment-report outputs/calibration-experiment/experiment.json --out outputs/quality-router-calibration --model-id ocrtech-tesseract-quality-router --primary-engine tesseract --secondary-engine tesseract --primary-engine-kwarg 'language=nep+eng' --primary-engine-kwarg 'psm=6' --secondary-engine-kwarg 'language=nep+eng' --secondary-engine-kwarg 'psm=13' +ocrtech calibrate-script-router --experiment-report outputs/script-calibration/experiment.json --out outputs/script-router-calibration --model-id ocrtech-tesseract-surya-script-router --primary-engine tesseract --secondary-engine surya --primary-engine-kwarg 'language=nep+eng' --primary-engine-kwarg 'psm=6' --secondary-engine-kwarg 'mode=auto' +ocrtech calibrate-tesseract-psm-ensemble --primary-experiment-report outputs/psm13-calibration/experiment.json --variant-experiment psm7=outputs/psm7-calibration/experiment.json --variant-experiment psm11=outputs/psm11-calibration/experiment.json --variant-experiment psm13=outputs/psm13-calibration/experiment.json --out outputs/psm-ensemble-calibration --model-id ocrtech-tesseract-psm-ensemble +ocrtech run-experiment --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl --out outputs/experiment --baselines candidate,tesseract,stock-paddle --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json +ocrtech review-claim --experiment outputs/internal-claim/experiment.json --experiment outputs/ocrbench-v2/experiment.json --config claim-review.json --model-config runs/recognizer/package/model-card.json --out outputs/claim-review +ocrtech validate-claim --benchmark-report outputs/bench/report.json --config validation.json --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json --out outputs/validation +``` + +`parse-doc` writes: + +- `document.json` +- `document.md` +- `tables/*.csv` +- `tables/*.html` +- `figures/*.png` +- `figures/metadata.json` +- `quality.json` + +Figure exports are explicit. If an engine supplies `figure.image_path`, +`parse-doc` copies that file into `figures/`. If an engine supplies only a +figure bbox and the source is a regular image or page-bundle directory, +`parse-doc` crops the region into `figures/{figure_id}.png` when Pillow is +available. For bundles, it uses `figure.page_index` to select the page image. If +the artifact cannot be created, `figures/metadata.json` records +`metadata.export_warnings` instead of silently dropping the figure image. + +When `--fallback-engine` or `--fallback-model-config` is configured, +`document.json` records `metadata.fallback` with the primary engine, fallback +engine/model card, confidence threshold, trigger reason, and success/error +outcome. Benchmarks surface this as diagnostic metrics such as +`fallback_triggered`, `fallback_succeeded`, and `fallback_failed`. +The same fallback flags are available on `benchmark` and `run-experiment` for +candidate/ours runs, and experiment metadata records the fallback model-card +path, SHA-256, and confidence threshold. + +## Engines + +`auto` tries deterministic sidecar OCR first, then PaddleOCR, then Tesseract. Optional engines fail loudly when binaries or packages are missing. + +Sidecar inputs are useful for tests and staged pipelines. For `page.png`, provide one of: + +- `page.png.ocr.json` +- `page.ocr.json` +- `page.png.txt` +- `page.txt` + +## Data + +Use `prepare-data` to verify actual fields and image paths before training. By default it fails on the first bad row. Use `--skip-invalid` when the upstream dataset is messy and you want a usable manifest plus an explicit reject log instead of a silent partial corpus. + +Use `merge-manifests` to assemble multi-source corpora without ad hoc `cat` pipelines, then `audit-manifest` before training and `split-manifest` to create deterministic leakage-safe train/eval manifests. The split groups exact sample signatures before assignment so duplicate image+text examples cannot land on both sides. + +Known Hugging Face adapters now stamp verified slice metadata automatically: + +- `gauravgiri/nepali-ocr-dataset`: `nepali`, `real` +- `prashant0919/nepali-synthetic-ocr-lines`: `nepali`, `synthetic` +- `nvidia/OCR-Synthetic-Multilingual-v1 --hf-subset en`: `english`, `synthetic`, `line_crop`, `en` + +Use `rebalance-manifest` only on the training split after `split-manifest`. Rebalancing before the split would turn deliberate duplication into leakage. + +Use `audit-recognizer-corpus` before launching a recognizer training job. It +checks the actual train/eval manifests for sample counts, Latin-script coverage, +Devanagari coverage, mixed-script examples, real/synthetic training coverage, +and train/eval leakage. This guardrail was added after the failed PaddleOCR +mixed v4 branch: a new GPU run should not start unless English preservation and +Nepali coverage are explicit in the corpus. + +Use `build-recognizer-corpus` to assemble and upsample the training manifest by +actual text buckets, not by wishful dataset labels. It tracks `latin_only`, +`devanagari_only`, `mixed_devanagari_latin`, `real`, and `synthetic` coverage, +then writes a build JSON/Markdown summary next to the output manifest. It does +not invent missing categories: if a required bucket is absent from the input +manifests, the command fails so you know to add real source data before using +GPU time. + +`train-recognizer --backend paddleocr` writes +`recognizer-training-summary.json` alongside the generated PaddleOCR config. +Keep that file with any remote/GPU training bundle and pass it to +`package-recognizer --training-summary`; it records manifest hashes, slice +counts, character coverage, label/config hashes, and launch hyperparameters. +`bundle-recognizer-training` records the bundled summary path and SHA-256 under +`training_summary`, and packaged recognizer model cards record the same +provenance when supplied. For real remote/GPU runs, use +`bundle-recognizer-training --require-corpus-audit`; it runs the recognizer +corpus audit and refuses to create the bundle if the corpus does not meet the +configured English, Devanagari, mixed-script, real/synthetic, and leakage gates. +The gate also checks isolated Latin-only and Devanagari-only samples; a corpus +made entirely of mixed-script rows is not enough to protect English preservation +or Nepali-only recognition. + +Planned source datasets: + +- `gauravgiri/nepali-ocr-dataset` +- `prashant0919/nepali-synthetic-ocr-lines` +- `nvidia/OCR-Synthetic-Multilingual-v1` +- `cfilt/RoundTripOCR-nepali` + +The verified Hugging Face adapters are intentionally dataset-specific where the upstream schema is non-standard: + +- `gauravgiri/nepali-ocr-dataset`: generic `image` + `text` +- `prashant0919/nepali-synthetic-ocr-lines`: `metadata.jsonl` plus per-line image files, downloaded lazily for the requested rows +- `cfilt/RoundTripOCR-nepali`: correction pairs via `ocr` and `correct` +- `nvidia/OCR-Synthetic-Multilingual-v1`: HDF5 shards with `annotations.line_bboxes`; use `--hf-subset en` for English crops + +Some upstream corpora are dirty. For example, the current `gauravgiri/nepali-ocr-dataset` sample prefix includes null `text` rows, so use `--skip-invalid` when you want a prepared manifest plus an explicit reject log instead of a hard stop. + +## Baselines + +The benchmark command supports: + +- `ours` +- `tesseract` +- `stock-paddle` +- `glm-ocr` +- `paddleocr-vl` + +`glm-ocr` and `paddleocr-vl` are external command hooks: + +```bash +export OCRTECH_GLM_OCR_CMD='python scripts/run_glm_ocr.py --input {input} --out {out}' +export OCRTECH_PADDLEOCR_VL_CMD='python scripts/run_paddleocr_vl.py --input {input} --out {out}' +``` + +The command must include both `{input}` and `{out}` placeholders. It may print Markdown to stdout or write `{out}/document.md`; `ocrtech` now normalizes that Markdown into its own `document.json` schema so external VLM baselines can still be scored for text, table cells, and reading order. + +The bundled wrappers are thin adapters, not magic: + +- `scripts/run_glm_ocr.py` uses the official `transformers` path from the `zai-org/GLM-OCR` model card. +- `scripts/run_paddleocr_vl.py` uses the official `PaddleOCRVL(...).predict(...)` path from the `PaddlePaddle/PaddleOCR-VL-1.6` model card. +- GLM-OCR still expects a git `transformers` build upstream. On `k2`, the released `transformers 4.57.6` package was not enough to instantiate its processor classes; a dedicated Python 3.11 env with git `transformers` and `torchvision` was required before the wrapper would run. +- PaddleOCR-VL still needs the OCR extras that the official docs call out, for example `pip install "paddlex[ocr]==3.7.1"`. + +## Validation Before SOTA Claims + +Use `validate-claim` before making any quality claim. The default validator is intentionally strict and will fail unless the benchmark report has held-out sample coverage, required slices, leakage checks, paired candidate-vs-baseline gates, and per-gate evidence such as paired wins and optional significance thresholds. + +Use `recognizer-admission-decision` after a recognizer admission validation run. +It writes a small JSON/Markdown decision artifact: `admit` only for +`claim_status=validated` with `passed=true`, otherwise `reject` with failed +gate metrics and next actions for the next model branch. + +Use `create-eval-pack` for internal mixed Nepali/English document tests and `audit-eval-pack` to check whether the pack is claim-ready. The internal pack now includes a real `multi_page` slice in addition to the single-page paragraph, table, form, and figure templates. Text packs are for pipeline smoke tests only; image packs are required for OCR claims. + +Use `preflight-claim` before `run-experiment` on any claim-grade run. It fails early on missing references, invalid model cards, missing baseline dependencies, non-claim-ready eval packs, bad validation configs, and train/eval leakage. + +For routed or thresholded candidates such as `quality_select_composite` and `script_select_composite`, claim-grade runs also require explicit calibration provenance in the model card. If the candidate was calibrated on the same eval manifest you are about to benchmark, `preflight-claim` now fails on purpose. + +For those routed candidates, claim-grade runs also need the raw component baselines in the same benchmark. If a `script_select_composite` candidate is built from Tesseract and Surya, the claim run must include `candidate,tesseract,surya`; otherwise the repo now fails preflight because distinctness cannot be checked honestly. + +Use `run-experiment` to benchmark every manifest input, write reproducibility metadata, and run claim validation in one output directory. + +Use `review-claim` after the internal pack and public benchmark suites have all been run. It audits the portfolio-level evidence bundle so one green experiment does not get misrepresented as a broad SOTA claim. + +Use a candidate model card for any real benchmark of `ours`; see [docs/MODELS.md](docs/MODELS.md). +Use `filter-manifest` when a baseline is too expensive to run on the full pack and you need a reproducible exploratory subset instead of hand-editing a JSONL. + +See [docs/PREFLIGHT.md](docs/PREFLIGHT.md), [docs/CLAIM_REVIEW.md](docs/CLAIM_REVIEW.md), [docs/EVAL_PACK.md](docs/EVAL_PACK.md), [docs/MODELS.md](docs/MODELS.md), [docs/EXPERIMENTS.md](docs/EXPERIMENTS.md), and [docs/VALIDATION.md](docs/VALIDATION.md). + +## Remote Training + +Use `docs/REMOTE_TRAINING.md` for the current `k2` host notes, `audit-remote-host`, and the safe SSH workflow. Do not commit SSH passwords or machine-specific secrets. + +## Current HF Result + +The strongest verified TrOCR candidate so far is the rebalanced mixed-script run trained on `k2` and evaluated on a held-out `180`-sample mixed Nepali/English line set. + +That run is materially better than our own earlier HF candidates: + +- overall benchmark CER: `1.628157` -> `0.586139` +- English CER: `0.379729` -> `0.300454` +- Nepali CER: `7.870295` -> `2.014560` +- real-slice CER: `12.702701` -> `1.770016` + +But the first named baseline comparison is still bad news for the project claim: on that same eval split, `tesseract` is better than the rebalanced candidate on overall, English, Nepali, real, and synthetic OCR metrics. + +The next attempted rescue path also failed. On June 14, 2026, a ByT5 text-correction model trained on `k2` from mined Tesseract noisy/clean pairs was packaged as a `text_correction_composite` candidate with raw Tesseract as the base engine. On the same held-out `180`-sample eval, that composite regressed catastrophically: overall CER `6.912065` versus raw Tesseract `0.329012`, with paired CER `0` wins, `170` losses, and `10` ties. That rules out the current Tesseract-plus-corrector path as a serious answer. + +There is one narrower improvement worth keeping, but it is not a claim-grade win. A routed Tesseract composite now exists as `quality_select_composite`: it runs default Tesseract (`psm=6`) and a secondary variant (`psm=13`), scores both outputs with the local `quality_score`, and switches only when the secondary beats the primary by a configured margin. + +The same-pack tuned run looked slightly better on the original `180`-sample mixed eval, but that was not honest enough. After splitting that pack into a separate calibration half (`89` samples) and untouched holdout half (`91` samples), the calibrated routed candidate still improved overall CER on the holdout, just by much less: + +- raw tesseract holdout overall CER: `0.342234` +- routed tesseract holdout overall CER: `0.340608` +- paired CER on holdout: `2` wins, `1` loss, `88` ties + +That makes it a plausible engineering baseline, not a frontier result. The effect size is tiny, the pack still is not claim-ready, and the real Devanagari recognizer problem remains open. + +We also tested a richer calibrated Tesseract ensemble over `psm=6/7/11/13` with learned quality-score biases. That looked better on the calibration half, but it failed on the untouched holdout: overall CER regressed from `0.342234` to `0.358352`. That is exactly the kind of result the new calibration workflow is supposed to expose. + +The more useful June 14, 2026 result is a calibrated Tesseract-plus-Surya script router on `k2`. Raw Surya was not competitive overall on the `91`-sample holdout (`2.198362` CER vs raw Tesseract `0.342234`), but it was materially better on the Nepali slice. A routed `script_select_composite` candidate calibrated on the separate `89`-sample half and evaluated on the untouched `91`-sample holdout did better than raw Tesseract across every reported slice: + +- raw tesseract holdout overall CER: `0.342234` +- calibrated script router overall CER: `0.313084` +- paired CER on holdout: `9` wins, `0` losses, `82` ties + +That is the strongest verified baseline improvement in the repo so far. It is still not a frontier claim, because it is one internal line-level split and not the full claim bundle, but it is a real out-of-sample gain and a better direction than the current TrOCR branch. The router now also supports `routing_granularity=page` for mixed multi-page documents; the calibrated `k2` result above is still the original document-level calibration, not a new page-level benchmark. + +We also ran an exploratory page-level script router on the same `91`-sample holdout, reusing the document-level threshold `0.4`. The result was identical to the document-level router (`0.313084` overall CER, `9` wins, `0` losses, `82` ties vs raw Tesseract). That is expected for this pack because it is dominated by single-page line-crop samples; page routing needs a real multi-page document pack before it can prove anything new. + +The internal image eval pack now has a real rendered `multi_page` slice with bundled page images and actual drawn table grids. On that small rendered smoke pack, the plain script router already beat Tesseract strongly on text (`0.243224` vs `0.540584` overall CER) and on the dedicated table page (`table_cell_f1` `1.0` vs `0.0`). A follow-up exploratory model with `secondary_structure_backfill=true` kept the same CER and backfilled Surya table structure on the multi-page report, lifting aggregate `table_cell_f1` from `0.5` to `1.0` on that smoke pack. That is still not claim-grade evidence because the pack is tiny and only `clean`, but it is a real document-capability improvement. + +The next step was to stop calling it smoke evidence and run the same backfill router on a claim-ready rendered image pack. That broader internal pack contains `25` samples across `clean`, `scan`, `blur`, `low_contrast`, and `skew`, and its audit passed cleanly. The calibrated backfill router also passed `preflight-claim` on that pack. Against raw Tesseract on the same rendered claim pack it still held up on text and structure: + +- overall CER: `0.248654` vs `0.689214` +- blur CER: `0.260039` vs `0.991613` +- scan CER: `0.237424` vs `0.593631` +- skew CER: `0.239317` vs `0.766067` +- aggregate `table_cell_f1`: `0.955556` vs `0.0` +- aggregate `table_exact_match`: `0.8` vs `0.0` +- paired CER: `9` wins, `1` loss, `15` ties + +That result is good enough to keep and bad enough to describe plainly. The multi-page slice tied Tesseract on CER (`0.315789` each) while still recovering table structure that Tesseract missed (`table_cell_f1` `0.911111` vs `0.0`). The form slice regressed slightly on CER (`0.231933` vs `0.230010`). Aggregate reading-order pair accuracy on this rendered pack is also still worse than Tesseract (`0.764` vs `0.816`). So the current candidate is meaningfully better on OCR robustness and table extraction for this internal rendered pack, but it is not yet a clean document-understanding leader. + +We then widened the comparison beyond Tesseract. Two more June 14, 2026 checks on `k2` matter: + +- raw Surya on the same `25`-sample rendered claim pack is better on text CER than the calibrated backfill router: `0.195260` vs `0.248654` +- but the router is still not pointless: it ties Surya on aggregate `table_cell_f1` (`0.955556` each) and is slightly better on aggregate `reading_order_pair_accuracy` (`0.764000` vs `0.750909`) + +That means the current routed candidate is still not “better than what is out there” on this rendered document pack. Raw Surya is the stronger text OCR baseline here. The router is only defending a narrower position: it preserves comparable structure while staying closer to the existing Tesseract-centered stack. + +We also fixed a real integration bug in the local Paddle baseline. `paddleocr 3.7.0` on `k2` returns v3 `OCRResult` objects, and the old adapter in this repo treated them like the older tuple/list API and silently dropped their text. After fixing that parser and installing `paddlepaddle 3.3.1`, `stock-paddle` became a real benchmark target on `k2`. On this same rendered pack it is still a weak baseline overall: + +- stock-Paddle overall CER: `1.749081` +- stock-Paddle aggregate `table_cell_f1`: `0.0` +- stock-Paddle aggregate `reading_order_pair_accuracy`: `1.0` + +So the current picture is sharper now, not prettier: the routed candidate beats Tesseract and this stock-Paddle configuration on OCR robustness and table extraction, but it still loses to raw Surya on text CER. + +We then used the new `filter-manifest` helper to run a broader but still tractable rendered-document comparison on `k2`: a `5`-sample blur-only subset spanning paragraph, table, form, figure, and multi-page layouts. All `30` runs completed without error across `candidate`, `tesseract`, `surya`, `stock-paddle`, `glm-ocr`, and `paddleocr-vl`. The result sharpened the ranking again: + +- candidate CER: `0.223273` +- raw Surya CER: `0.197601` +- Tesseract CER: `0.732364` +- stock-Paddle CER: `2.810426` +- GLM-OCR CER: `2.820269` +- PaddleOCR-VL CER: `18.470663` + +On that subset, the current backfill candidate still held the structure line where Tesseract and the weaker VLM baselines collapsed (`table_cell_f1 = 1.0` for the candidate versus `0.0` for Tesseract, stock-Paddle, GLM-OCR, and PaddleOCR-VL). But it still did not beat raw Surya on text CER. That keeps the same honest conclusion intact: on rendered document OCR text, the strongest baseline in this repo is still Surya, not the current routed candidate. + +The stricter rendered-document result is now explicit too. On `k2`, a larger rendered `claimpack-rendered-v2` split was calibrated separately and evaluated on an untouched `50`-sample holdout. That calibration selected `primary_script_threshold=0.0`, so the routed `script_select_composite` collapsed to "always pick Surya" on the calibration set. On the holdout: + +- candidate CER: `0.180893` +- raw Surya CER: `0.180893` +- candidate `table_cell_f1`: `0.947500` +- raw Surya `table_cell_f1`: `0.970833` + +That is the ceiling of the current branch, not a win. The candidate no longer even has a text-vs-structure tradeoff story there; on this larger rendered holdout it effectively degenerates to Surya on text and still loses slightly on table structure. The claim guardrails now reject this case directly: routed candidates that collapse to one engine across all calibration samples fail `candidate-calibration` in both `preflight-claim` and `review-claim`. + +We also tested whether a quality-based Surya/Tesseract selector could recover anything on that same rendered-v2 setup. It did not. The rendered-v2 Surya quality router calibrated to `selected_threshold=0.0`, then landed at: + +- candidate CER: `0.181128` +- raw Surya CER: `0.180893` +- candidate `table_cell_f1`: `0.947500` +- raw Surya `table_cell_f1`: `0.947500` +- paired CER vs Surya: `0` wins, `1` loss, `49` ties + +That is not a distinct better engine. It is basically Surya with one extra mistake. + +We then tested `line_align_composite` on the same rendered-v2 holdout: keep Surya text, align it to Tesseract's line scaffold, and see whether the ordering gain is worth the merge. It was not: + +- candidate CER: `0.310549` +- raw Surya CER: `0.180893` +- candidate `table_cell_f1`: `0.897500` +- raw Surya `table_cell_f1`: `0.970833` +- candidate `reading_order_pair_accuracy`: `0.792727` +- raw Surya `reading_order_pair_accuracy`: `0.750909` +- paired CER vs Surya: `0` wins, `7` losses, `43` ties + +That is another useful paper result, but not a winning model branch. Better line ordering alone did not rescue the Surya/Tesseract composite family. + +The upper bound is now measured too. A sample-level oracle that chooses the better of raw Surya and raw Tesseract per holdout sample using ground-truth CER landed at: + +- oracle CER: `0.180529` +- raw Surya CER: `0.180893` +- raw Tesseract CER: `0.714207` +- oracle choices: Surya `49`, Tesseract `1` +- paired CER vs Surya: `1` win, `0` losses, `49` ties + +That closes the routing branch for text OCR. There is almost no remaining sample-level headroom in the Surya/Tesseract component family on rendered-v2. The next credible model-side step has to use a genuinely stronger recognizer or document model, not another selector over these same two engines. + +See [docs/EXPERIMENTS.md](docs/EXPERIMENTS.md) for the exact commands, artifact paths, and the current evidence gap. + +If you are turning this work into a paper, start from [docs/PAPER_NOTES.md](docs/PAPER_NOTES.md). Then use [docs/PAPER_RESULTS.md](docs/PAPER_RESULTS.md) for the current generated comparison tables and branch chronology tied to local artifacts. + +## References + +- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) +- [PP-StructureV3 docs](https://paddlepaddle.github.io/PaddleOCR/latest/version3.x/pipeline_usage/PP-StructureV3.html) +- [PaddleX table structure recognition docs](https://paddlepaddle.github.io/PaddleX/3.4/en/module_usage/tutorials/ocr_modules/table_structure_recognition.html) +- [PaddleOCR-VL-1.6](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) +- [Tesseract docs](https://tesseract-ocr.github.io/docs/) +- [GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) +- [Surya OCR](https://github.com/datalab-to/surya) diff --git a/ocr-tech-mps-integration/RESEARCH.md b/ocr-tech-mps-integration/RESEARCH.md new file mode 100644 index 0000000000000000000000000000000000000000..af27b010a57d2e4cf52ed5850253512aa790d3ab --- /dev/null +++ b/ocr-tech-mps-integration/RESEARCH.md @@ -0,0 +1,375 @@ +# Research State: Nepal Language Pack OCR + +Updated: 2026-08-09 + +## Current Goal + +The end goal is an OCR system covering the multiple languages of the Nepal pack, +built out of Paddle, **with Limbu and Newar (Nepal Bhasa) as the first two focus +languages** (user steer, 2026-07-04). The system should accept real book/page images +or PDFs for any language in the pack and output editable Unicode text with page, +line, crop, confidence, and provenance evidence preserved. Limbu stays the proven +template — it has the strongest recognizer and the deepest harvest. Newar is the +deliberate second: it has its own native script (Prachalit), real cultural demand, +and is currently the weakest trained recognizer, so it exercises every part of the +method that Limbu already proved. Every stage that was built Limbu-specific must +generalize: harvest, silver-label extraction, recognizer training, script routing, +page/book OCR, and eval. The product is the full pipeline: page +detection, script-aware routing to per-script recognizers, target-region and +furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and +review artifacts for hard lines. YOLO line detection stays a diagnostic/probe; +Paddle-first is the production story. + +## The Language Pack + +Two inventories exist and they are not the same thing: + +- The registry `languages/nepal-v1.json` tracks 18 target languages (17 Nepal + English) + with scripts, fonts, dictionaries, and corpus status. +- The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found + **50 normalized languages (53 raw labels) in 400 articles over ~7 months** (see the + harvest section). Labels include all registry languages plus Chamling Rai, Bahing, + Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo, + Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi, + Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more. + Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा, + ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) — + `tools/build_gorkhapatra_language_pack_inventory.py` owns that normalization and the + label→registry mapping. + +Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded +2026-06-18 scorecard in `docs/RECOGNIZER_GOAL_PROGRESS.md`): + +- **Limbu/Sirijonga 0.041 real-print** (champion: + `runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23`, + cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval). +- **8 Devanagari languages 0.075–0.129 real-print** (Majhi, Magar, Sunuwar-Deva, Magahi, + Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer. +- **Sunuwar (Jenticha) 0.121 real-crop** after the same-font retrain (0.342 synth-only). +- **Tirhuta 0.343 and Newa 0.613 synthetic-only** — weak, transliteration-fed, honest + fails; Newa needs real printed-register Nepal Lipi, not more pixels. +- **Kirat Rai (70 real crops) and Lepcha (132 real crops)** blocked on legacy-font + converter coverage outside their eval editions; **Gurung Khema** has no real print + anywhere (revival script) — synthetic-only by nature, claim-scoped accordingly. + +The near-term pack claim should therefore be script-family shaped: one Sirijonga +recognizer, one shared Devanagari recognizer covering the many Devanagari-written +languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa, +Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval +crops exist. + +**Focus order (2026-07-04 steer): Limbu first, Newar second.** Newar state: the +Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own +synthetic eval; transliteration-fed training failed; real printed-register Nepal +Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar +route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20 +Gorkhapatra pages; `newar` added to the registry (Devanagari primary + Prachalit +secondary; audit passes); a frozen by-date pack built at +`outputs/newar-devanagari-silver-pack-v1/` (train 4,096 / dev 1,528 / test 1,139 +lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently +verified; pack sha256 +`1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85`); and the +**first real-print Devanagari-Newar baseline** was scored locally (CPU export +route, `tools/score_recognizer_pack.py`, per `docs/LOCAL_RECOGNIZER_SCORING.md`): +the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at +**codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158** (n=1,139 silver +lines — machine-converted labels, diagnostic, not claim-grade). The same-day +warm-start fine-tune (**newar-ft-v1**: deva-v2 base, LR 2e-5 cosine, 60 epochs, +batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance +`ashish-experiments`, ~68 min; artifacts + train log at +`outputs/aws-newar-deva-ft-v1/`) scores the same frozen test at **codepoint CER +0.0465, grapheme CER 0.0955, exact-line 0.245** — a 25/27% relative CER drop and ++55% exact-line over the deva-v2 baseline, scored through the identical local +CPU path. **Newar's grapheme CER now sits inside deva-v2's own-language range +(0.075–0.129): the second focus language reached Devanagari-pack parity from +zero in one day.** Same claim boundary: silver machine-converted labels, +diagnostic, not claim-grade until native review. Next Newar lever: native +Prachalit (real printed-register data), and page-level Newar eval for the +region-filtering track. + +## Source Rule + +For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata +only. Use `https://gorkhapatraonline.com/publications/naya-nepal` to find the language, +date, and article, but do not use the blog/card image for OCR, training, or evaluation +when the full paper exists. The authoritative source is the matching Gorkhapatra epaper +PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus +the text layer extracted from that same PDF page, with PDF URL/hash, page number, +rendered image hash, extraction command/version, font evidence, and converter/map hash +recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with +`src/ocrtech/limbu_legacy.py` + `data/mappings/limbu-legacy/Limbu.map`. The same +pattern — classify the text layer, then build an explicit converter only when the font +requires it — replicates to every other language. Manual transcription is the fallback +for no usable text layer, not the default. + +## Getting More Gorkhapatra Pages (the specified harvest recipe) + +All numbers below were verified live on 2026-07-04. + +**Archive size.** The Naya Nepal publication archive paginates at +`https://gorkhapatraonline.com/publications/naya-nepal?page=N` with 5 articles per +page and runs to page ~352 before drying up: **~1,750 language-page articles total** +(about 2 per day going back years). The 60-page crawl already yielded 300 items across +53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly +older dates. + +**Epaper endpoint behavior (the critical constraint).** The date-range endpoint +`https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra` +**caps results at 30 PDFs per query and ignores `page=N` pagination**. A wide window +silently drops most dates — the 2026-07-04 six-window discovery pass aligned only +111/300 items for exactly this reason, with unmatched dates interleaved among matched +ones. **Seven-day windows return complete daily coverage (7/7 verified).** Always +harvest with weekly windows. + +**The recipe** (one command; add one `--epaper-url` per week to cover the publication +date span being harvested): + +```bash +env PYTHONPATH=src uv run --python 3.11 --with pymupdf \ + ocrtech audit-gorkhapatra-language-pages \ + --out outputs/gorkhapatra-language-pages-full- \ + --publication-url https://gorkhapatraonline.com/publications/naya-nepal \ + --publication-pages 80 \ + --epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \ + --epaper-url '...one per week, going back...' \ + --download-pdfs \ + --render-pages \ + --timeout-seconds 180 +``` + +The command aligns every publication card (all languages at once — alignment is +language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL), +searches pages for the language-page markers (`भाषा पृष्ठ` and its legacy-encoding +form `efiff k[i7`), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly +windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with +**400 publication items, 195 aligned dates, 394 rendered language-page hits, zero +warnings** at `outputs/gorkhapatra-language-pages-full-2026-07-04/`. Batch 2 +(2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more +items: **798 items, 294 aligned dates, 589 further page hits**, at +`outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/`. The merged normalized +inventory (`tools/build_gorkhapatra_language_pack_inventory.py`) shows **56 +languages, 983 rendered language-page hits, most languages with ~20–27 pages +spanning 14 months**, at `outputs/language-pack-inventory-merged-2026-07-04/`. For +comparison, the entire previous state of the project had 8 usable pages for one +language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least +March 2024 — verified the epaper endpoint serves complete weeks there) is another +~2× and stays month-batchable with the same recipe. + +**Scaling to the full archive.** Full depth is ~700+ unique dates ≈ 35 GB of PDFs. +Do not do it in one run: batch by month (4–5 weekly windows + the matching +publication-page depth), run `extract-gorkhapatra-language-page-pdf-text` per batch, +then keep rendered pages, extracted text, and hashes; PDFs can be pruned after +extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume +(`cdjk@100.110.148.43:/Volumes/TRANSCEND`) is the right home for a full-depth mirror. + +**Per-language downstream.** After harvest, run per language: + +```bash +env PYTHONPATH=src uv run --python 3.11 --with pymupdf \ + ocrtech extract-gorkhapatra-language-page-pdf-text \ + \ + --candidate-language