Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- ocr-tech-mps-integration/.gitignore +19 -0
- ocr-tech-mps-integration/.ignore +9 -0
- ocr-tech-mps-integration/CLAUDE.md +38 -0
- ocr-tech-mps-integration/README.md +371 -0
- ocr-tech-mps-integration/RESEARCH.md +375 -0
- ocr-tech-mps-integration/configs/cdjk-language-corpus-migration-v1.json +170 -0
- ocr-tech-mps-integration/configs/claim-review-sota-v1.json +121 -0
- ocr-tech-mps-integration/configs/eval-harness-recipes-v1.json +651 -0
- ocr-tech-mps-integration/configs/eval-target-coverage-v1.json +193 -0
- ocr-tech-mps-integration/configs/limbu-devanagari-ppocrv5-admission-plan-v1.json +68 -0
- ocr-tech-mps-integration/configs/limbu-first-ocr-pipeline-v1.json +0 -0
- ocr-tech-mps-integration/configs/limbu-sirijonga-augmentation-ablation-v1.json +68 -0
- ocr-tech-mps-integration/configs/namdhinggo-font-assets-v1.json +106 -0
- ocr-tech-mps-integration/configs/nepal-font-assets-v1.json +110 -0
- ocr-tech-mps-integration/configs/nepal-language-pack-pipeline-v1.json +1051 -0
- ocr-tech-mps-integration/configs/nepali-table-name-lexicon-v1.txt +10 -0
- ocr-tech-mps-integration/configs/recognizer-eval-pack-normalization-v1.json +165 -0
- ocr-tech-mps-integration/configs/validation-complex-table-surya.json +57 -0
- ocr-tech-mps-integration/configs/validation-hard15-surya.json +59 -0
- ocr-tech-mps-integration/configs/validation-hf-recognizer-next-v1-admission.json +79 -0
- ocr-tech-mps-integration/configs/validation-internal-sota-v1.json +120 -0
- ocr-tech-mps-integration/configs/validation-paddle-hard50-tesseract.json +52 -0
- ocr-tech-mps-integration/configs/validation-public-omnidocbench-sota-v1.json +60 -0
- ocr-tech-mps-integration/configs/validation-public-omnidocbench-surya.json +46 -0
- ocr-tech-mps-integration/configs/validation-public-real5-sota-v1.json +67 -0
- ocr-tech-mps-integration/configs/validation-table-ledger-surya.json +39 -0
- ocr-tech-mps-integration/data/dictionaries/devanagari-v1.txt +141 -0
- ocr-tech-mps-integration/data/dictionaries/gurungkhema-v1.txt +58 -0
- ocr-tech-mps-integration/data/dictionaries/kiratrai-v1.txt +58 -0
- ocr-tech-mps-integration/data/dictionaries/lepcha-v1.txt +74 -0
- ocr-tech-mps-integration/data/dictionaries/limbu-sirijonga-v1.txt +76 -0
- ocr-tech-mps-integration/data/dictionaries/limbu-sirijonga-v2.txt +91 -0
- ocr-tech-mps-integration/data/dictionaries/limbu-sirijonga-v3.txt +98 -0
- ocr-tech-mps-integration/data/dictionaries/newa-clean-v1.txt +103 -0
- ocr-tech-mps-integration/data/dictionaries/newa-htr-v1.txt +177 -0
- ocr-tech-mps-integration/data/dictionaries/newa-native-v1.txt +163 -0
- ocr-tech-mps-integration/data/dictionaries/newa-v1.txt +97 -0
- ocr-tech-mps-integration/data/dictionaries/sunuwar-v1.txt +44 -0
- ocr-tech-mps-integration/data/dictionaries/tirhuta-v1.txt +82 -0
- ocr-tech-mps-integration/data/mappings/limbu-legacy/Limbu.map +146 -0
- ocr-tech-mps-integration/data/mappings/limbu-legacy/README.txt +1 -0
- ocr-tech-mps-integration/docs/ACTIVE_RESEARCH_CHECKLIST.md +386 -0
- ocr-tech-mps-integration/docs/CDJK_LANGUAGE_CORPUS_MIGRATION.md +310 -0
- ocr-tech-mps-integration/docs/CLAIM_REVIEW.md +333 -0
- ocr-tech-mps-integration/docs/CLAUDE_HANDOFF.md +613 -0
- ocr-tech-mps-integration/docs/CODEX_HANDOFF.md +0 -0
- ocr-tech-mps-integration/docs/DATASETS.md +381 -0
- ocr-tech-mps-integration/docs/EVAL_HARNESS_GOAL.md +437 -0
- ocr-tech-mps-integration/docs/EVAL_HARNESS_RESEARCH_GOAL.md +111 -0
- ocr-tech-mps-integration/docs/EVAL_HARNESS_RUNBOOK.md +803 -0
ocr-tech-mps-integration/.gitignore
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.venv/
|
| 2 |
+
__pycache__/
|
| 3 |
+
*.py[cod]
|
| 4 |
+
.pytest_cache/
|
| 5 |
+
.ruff_cache/
|
| 6 |
+
.mypy_cache/
|
| 7 |
+
|
| 8 |
+
data/*
|
| 9 |
+
!data/dictionaries/
|
| 10 |
+
!data/dictionaries/*.txt
|
| 11 |
+
outputs/
|
| 12 |
+
runs/
|
| 13 |
+
*.egg-info/
|
| 14 |
+
|
| 15 |
+
.DS_Store
|
| 16 |
+
|
| 17 |
+
spaces/limbu-ocr/models/
|
| 18 |
+
incoming/
|
| 19 |
+
logs/
|
ocr-tech-mps-integration/.ignore
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.venv/
|
| 2 |
+
__pycache__/
|
| 3 |
+
*.py[cod]
|
| 4 |
+
.pytest_cache/
|
| 5 |
+
|
| 6 |
+
data/
|
| 7 |
+
outputs/
|
| 8 |
+
runs/
|
| 9 |
+
*.egg-info/
|
ocr-tech-mps-integration/CLAUDE.md
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ocr-tech — agent instructions
|
| 2 |
+
|
| 3 |
+
## Drive documents
|
| 4 |
+
|
| 5 |
+
- `RESEARCH.md` is the research state and drive document. Read it first every
|
| 6 |
+
session; keep it current when results land.
|
| 7 |
+
- End goal: OCR covering the whole Nepal language pack, **Limbu first, Newar
|
| 8 |
+
(Nepal Bhasa) second** (steer 2026-07-04).
|
| 9 |
+
|
| 10 |
+
## Model delegation policy (steer 2026-07-04)
|
| 11 |
+
|
| 12 |
+
- **Fable (lead session model)**: orchestration, the hardest reasoning, design
|
| 13 |
+
decisions, and final verification of all delegated work. Best model available —
|
| 14 |
+
do not delegate judgment calls.
|
| 15 |
+
- **Codex**: treat as a peer, not a subordinate. It has good independent ways of
|
| 16 |
+
figuring things out. Hand it substantial, well-scoped build/diagnosis/research
|
| 17 |
+
tasks (new tools, refactors, root-cause hunts) with verified file paths and
|
| 18 |
+
acceptance criteria; always verify its output independently (run its tests,
|
| 19 |
+
spot-check its data).
|
| 20 |
+
- **Opus**: near-par with the lead model but slightly weaker; usable for hard
|
| 21 |
+
tasks and second opinions when a peer-level check is wanted.
|
| 22 |
+
- **Sonnet**: tasks that require no big thought — mechanical edits, bulk file
|
| 23 |
+
wrangling, running known command recipes, simple searches and summaries.
|
| 24 |
+
|
| 25 |
+
## Working rules
|
| 26 |
+
|
| 27 |
+
- Never assume file paths, schemas, or endpoint behavior — verify on disk / live
|
| 28 |
+
before coding against them (this repo's history has multiple wrong-assumption
|
| 29 |
+
scars; see RESEARCH.md "Mistakes To Avoid").
|
| 30 |
+
- Real CER comes from `tools/eval_metrics.py` (micro CER, codepoint + grapheme).
|
| 31 |
+
Never report Paddle `norm_edit_dis` as CER.
|
| 32 |
+
- Silver labels are machine-converted; they are training/diagnostic substrate,
|
| 33 |
+
not claim-grade ground truth. Claim-grade numbers need frozen, human-reviewed
|
| 34 |
+
eval packs.
|
| 35 |
+
- Long downloads/runs: launch detached (`nohup`), not under tool timeouts; watch
|
| 36 |
+
with polling loops. `ocrtech audit-gorkhapatra-language-pages` has no resume.
|
| 37 |
+
- Python runtime for Paddle work is 3.11 (`uv run --python 3.11`); silver tooling
|
| 38 |
+
needs `--with pymupdf --with npttf2utf`.
|
ocr-tech-mps-integration/README.md
ADDED
|
@@ -0,0 +1,371 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ocr-tech
|
| 2 |
+
|
| 3 |
+
Hybrid Nepali + English OCR and document-structure extraction toolkit.
|
| 4 |
+
|
| 5 |
+
This repository is a runnable foundation for the document-understanding plan:
|
| 6 |
+
|
| 7 |
+
- local OCR/structure pipeline first
|
| 8 |
+
- optional PaddleOCR/Tesseract/remote model adapters
|
| 9 |
+
- stable CLI commands for data prep, training recipes, parsing, and benchmarks
|
| 10 |
+
- explicit output schema for Markdown, JSON, tables, and figures
|
| 11 |
+
|
| 12 |
+
It does not claim that the fine-tuned recognizer is already trained. Training commands generate checked manifests, PaddleOCR label files, dictionaries, and recipe configs so long jobs can be run on a suitable host.
|
| 13 |
+
|
| 14 |
+
The active research steering checklist is tracked in
|
| 15 |
+
[docs/ACTIVE_RESEARCH_CHECKLIST.md](docs/ACTIVE_RESEARCH_CHECKLIST.md). It
|
| 16 |
+
records the cdjk remote synthesis leads, Gorkhapatra validation path, and paper
|
| 17 |
+
claim gates.
|
| 18 |
+
|
| 19 |
+
## Install
|
| 20 |
+
|
| 21 |
+
```bash
|
| 22 |
+
python3 -m venv .venv
|
| 23 |
+
. .venv/bin/activate
|
| 24 |
+
python -m pip install -e '.[dev]'
|
| 25 |
+
```
|
| 26 |
+
|
| 27 |
+
Optional integrations:
|
| 28 |
+
|
| 29 |
+
```bash
|
| 30 |
+
python -m pip install -e '.[eval]'
|
| 31 |
+
python -m pip install -e '.[datasets,engines]'
|
| 32 |
+
python -m pip install -e '.[hf-recognizer]'
|
| 33 |
+
```
|
| 34 |
+
|
| 35 |
+
The `eval` extra installs Pillow and PyMuPDF for image/PDF evaluation artifacts,
|
| 36 |
+
including Gorkhapatra epaper page rendering.
|
| 37 |
+
|
| 38 |
+
The `hf-recognizer` extra is the real dependency set for TrOCR-style training and inference here. It includes `torch`, `transformers`, `accelerate`, `sentencepiece`, `protobuf`, and `pillow`.
|
| 39 |
+
|
| 40 |
+
## CLI
|
| 41 |
+
|
| 42 |
+
```bash
|
| 43 |
+
ocrtech prepare-data labels.csv --out data/manifests --image-field image_path --text-field text
|
| 44 |
+
ocrtech prepare-data labels.csv --out data/manifests --image-field image_path --text-field text --skip-invalid
|
| 45 |
+
ocrtech prepare-data gauravgiri/nepali-ocr-dataset --hf --split train --limit 100 --out data/manifests
|
| 46 |
+
ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines --hf --split train --limit 100 --out data/manifests
|
| 47 |
+
ocrtech prepare-data nvidia/OCR-Synthetic-Multilingual-v1 --hf --hf-subset en --split test --limit 1000 --out data/manifests-english-test
|
| 48 |
+
ocrtech merge-manifests data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl --out data/manifests/nepali-mixed-all.jsonl
|
| 49 |
+
ocrtech audit-manifest data/manifests/labels-train.jsonl --require-slice nepali --require-slice english
|
| 50 |
+
ocrtech split-manifest data/manifests/labels-train.jsonl --out data/splits --eval-ratio 0.15
|
| 51 |
+
ocrtech rebalance-manifest data/splits/train.jsonl --out data/splits/train-rebalanced.jsonl --slice english --slice nepali
|
| 52 |
+
ocrtech audit-recognizer-corpus --train-manifest data/splits/train-rebalanced.jsonl --eval-manifest data/splits/eval.jsonl --out outputs/recognizer-corpus-audit
|
| 53 |
+
ocrtech build-recognizer-corpus data/manifests/english-lines.jsonl data/manifests/nepali-lines.jsonl data/manifests/mixed-crops.jsonl --out data/splits/recognizer-train-balanced.jsonl --target-latin-only 200 --target-devanagari-only 200 --target-mixed 100 --target-real 100 --target-synthetic 100
|
| 54 |
+
ocrtech materialize-references data/splits/eval.jsonl --out data/references --rewritten-manifest data/splits/eval-with-refs.jsonl
|
| 55 |
+
ocrtech create-eval-pack --out data/eval-pack --input-format image --degradation clean --degradation scan
|
| 56 |
+
ocrtech create-eval-pack --out data/eval-pack-table-figure-dev-v1 --input-format image --template table-ledger --template figure-caption --count-per-template 8 --variant-offset 100 --degradation clean --degradation scan --degradation blur --degradation low_contrast --degradation skew
|
| 57 |
+
ocrtech audit-eval-pack data/eval-pack/manifests/ocrtech-eval-pack.jsonl --require-claim-ready
|
| 58 |
+
ocrtech train-recognizer --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --out runs/recognizer
|
| 59 |
+
ocrtech train-recognizer --backend hf-vision-encoder-decoder --base-model microsoft/trocr-small-printed --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --out runs/hf-recognizer
|
| 60 |
+
ocrtech bundle-recognizer-training --recipe-dir runs/recognizer --train-manifest data/splits/train-rebalanced.jsonl --eval-manifest data/splits/eval.jsonl --out runs/recognizer-training-bundle --require-corpus-audit
|
| 61 |
+
ocrtech export-recognizer --training-config runs/recognizer/paddle_recognizer.yml --checkpoint runs/recognizer/checkpoints/best_accuracy --out runs/recognizer/export
|
| 62 |
+
ocrtech package-recognizer --inference-dir runs/recognizer/export/inference --dictionary runs/recognizer/nep_eng_dict.txt --out runs/recognizer/package --model-id ocrtech-nep-eng-rec-v1 --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --training-summary runs/recognizer/recognizer-training-summary.json --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
|
| 63 |
+
ocrtech package-hf-recognizer --model-dir runs/hf-recognizer/final-model --out runs/hf-recognizer/package --model-id ocrtech-hf-nep-eng-rec-v1 --base-model microsoft/trocr-small-printed --train-manifest data/splits/train.jsonl --eval-manifest data/splits/eval.jsonl --training-summary runs/hf-recognizer/recognizer-training-summary.json --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
|
| 64 |
+
ocrtech train-corrector --clean-manifest data/splits/train.jsonl --out runs/corrector
|
| 65 |
+
ocrtech train-corrector --hf-dataset cfilt/RoundTripOCR-nepali --hf-split train --limit 1000 --out runs/corrector-cfilt
|
| 66 |
+
ocrtech audit-model runs/recognizer/package/model-card.json
|
| 67 |
+
ocrtech audit-remote-host --host 100.97.238.103 --user k2 --password-env OCRTECH_REMOTE_PASSWORD --workdir '~/ocr-tech' --require-path '~/ocr-tech' --min-free-gb 20 --require-paddle-training --out outputs/remote-audit
|
| 68 |
+
ocrtech bootstrap-remote-host --host 100.97.238.103 --user k2 --password-env OCRTECH_REMOTE_PASSWORD --workdir '~/ocr-tech' --out outputs/remote-bootstrap
|
| 69 |
+
ocrtech sync-remote-workspace --host 100.97.238.103 --user k2 --password-env OCRTECH_REMOTE_PASSWORD --local-root . --remote-workdir '~/ocr-tech' --exclude .venv --exclude outputs --exclude runs --out outputs/remote-sync
|
| 70 |
+
ocrtech preflight-claim --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl --references-dir data/eval-pack/references --baselines candidate,tesseract,stock-paddle --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json --train-manifest data/splits/train.jsonl --out outputs/preflight
|
| 71 |
+
ocrtech parse-doc examples/sample-page.txt --out outputs/sample --engine sidecar
|
| 72 |
+
ocrtech parse-doc page.png --out outputs/page --engine candidate --model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json
|
| 73 |
+
ocrtech benchmark examples/sample-page.txt --out outputs/bench --baselines ours --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json
|
| 74 |
+
ocrtech summarize-benchmark --benchmark-report outputs/bench/report.json --out outputs/bench-summary --candidate ours --baseline tesseract --metric cer
|
| 75 |
+
ocrtech benchmark-error-analysis --benchmark-report outputs/bench/report.json --out outputs/bench-error-analysis --system surya --baseline tesseract --metric cer
|
| 76 |
+
ocrtech export-failure-manifest --source-manifest data/splits/eval.jsonl --top-failures outputs/bench-error-analysis/top-failures.json --out data/splits/eval-hard.jsonl --split hard_eval
|
| 77 |
+
ocrtech recognizer-admission-decision --validation-report outputs/validation-next-recognizer-hard50/validation.json --out outputs/recognizer-admission-next
|
| 78 |
+
ocrtech paper-benchmark-table --summary outputs/paper-claimpack-summary-tesseract/summary.json --summary outputs/paper-claimpack-summary-surya/summary.json --summary outputs/paper-claimpack-summary-stock-paddle/summary.json --out outputs/paper-claimpack-table --system candidate --system surya --system tesseract --system stock-paddle --label 'candidate=Candidate' --label 'surya=Surya' --label 'tesseract=Tesseract' --label 'stock-paddle=Stock Paddle' --source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' --source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' --source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' --source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' --metric cer --metric table_cell_f1 --metric reading_order_pair_accuracy --metric latency_seconds
|
| 79 |
+
ocrtech calibrate-quality-router --experiment-report outputs/calibration-experiment/experiment.json --out outputs/quality-router-calibration --model-id ocrtech-tesseract-quality-router --primary-engine tesseract --secondary-engine tesseract --primary-engine-kwarg 'language=nep+eng' --primary-engine-kwarg 'psm=6' --secondary-engine-kwarg 'language=nep+eng' --secondary-engine-kwarg 'psm=13'
|
| 80 |
+
ocrtech calibrate-script-router --experiment-report outputs/script-calibration/experiment.json --out outputs/script-router-calibration --model-id ocrtech-tesseract-surya-script-router --primary-engine tesseract --secondary-engine surya --primary-engine-kwarg 'language=nep+eng' --primary-engine-kwarg 'psm=6' --secondary-engine-kwarg 'mode=auto'
|
| 81 |
+
ocrtech calibrate-tesseract-psm-ensemble --primary-experiment-report outputs/psm13-calibration/experiment.json --variant-experiment psm7=outputs/psm7-calibration/experiment.json --variant-experiment psm11=outputs/psm11-calibration/experiment.json --variant-experiment psm13=outputs/psm13-calibration/experiment.json --out outputs/psm-ensemble-calibration --model-id ocrtech-tesseract-psm-ensemble
|
| 82 |
+
ocrtech run-experiment --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl --out outputs/experiment --baselines candidate,tesseract,stock-paddle --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json
|
| 83 |
+
ocrtech review-claim --experiment outputs/internal-claim/experiment.json --experiment outputs/ocrbench-v2/experiment.json --config claim-review.json --model-config runs/recognizer/package/model-card.json --out outputs/claim-review
|
| 84 |
+
ocrtech validate-claim --benchmark-report outputs/bench/report.json --config validation.json --candidate-model-config runs/recognizer/package/model-card.json --fallback-model-config runs/vlm-fallback/package/model-card.json --out outputs/validation
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
`parse-doc` writes:
|
| 88 |
+
|
| 89 |
+
- `document.json`
|
| 90 |
+
- `document.md`
|
| 91 |
+
- `tables/*.csv`
|
| 92 |
+
- `tables/*.html`
|
| 93 |
+
- `figures/*.png`
|
| 94 |
+
- `figures/metadata.json`
|
| 95 |
+
- `quality.json`
|
| 96 |
+
|
| 97 |
+
Figure exports are explicit. If an engine supplies `figure.image_path`,
|
| 98 |
+
`parse-doc` copies that file into `figures/`. If an engine supplies only a
|
| 99 |
+
figure bbox and the source is a regular image or page-bundle directory,
|
| 100 |
+
`parse-doc` crops the region into `figures/{figure_id}.png` when Pillow is
|
| 101 |
+
available. For bundles, it uses `figure.page_index` to select the page image. If
|
| 102 |
+
the artifact cannot be created, `figures/metadata.json` records
|
| 103 |
+
`metadata.export_warnings` instead of silently dropping the figure image.
|
| 104 |
+
|
| 105 |
+
When `--fallback-engine` or `--fallback-model-config` is configured,
|
| 106 |
+
`document.json` records `metadata.fallback` with the primary engine, fallback
|
| 107 |
+
engine/model card, confidence threshold, trigger reason, and success/error
|
| 108 |
+
outcome. Benchmarks surface this as diagnostic metrics such as
|
| 109 |
+
`fallback_triggered`, `fallback_succeeded`, and `fallback_failed`.
|
| 110 |
+
The same fallback flags are available on `benchmark` and `run-experiment` for
|
| 111 |
+
candidate/ours runs, and experiment metadata records the fallback model-card
|
| 112 |
+
path, SHA-256, and confidence threshold.
|
| 113 |
+
|
| 114 |
+
## Engines
|
| 115 |
+
|
| 116 |
+
`auto` tries deterministic sidecar OCR first, then PaddleOCR, then Tesseract. Optional engines fail loudly when binaries or packages are missing.
|
| 117 |
+
|
| 118 |
+
Sidecar inputs are useful for tests and staged pipelines. For `page.png`, provide one of:
|
| 119 |
+
|
| 120 |
+
- `page.png.ocr.json`
|
| 121 |
+
- `page.ocr.json`
|
| 122 |
+
- `page.png.txt`
|
| 123 |
+
- `page.txt`
|
| 124 |
+
|
| 125 |
+
## Data
|
| 126 |
+
|
| 127 |
+
Use `prepare-data` to verify actual fields and image paths before training. By default it fails on the first bad row. Use `--skip-invalid` when the upstream dataset is messy and you want a usable manifest plus an explicit reject log instead of a silent partial corpus.
|
| 128 |
+
|
| 129 |
+
Use `merge-manifests` to assemble multi-source corpora without ad hoc `cat` pipelines, then `audit-manifest` before training and `split-manifest` to create deterministic leakage-safe train/eval manifests. The split groups exact sample signatures before assignment so duplicate image+text examples cannot land on both sides.
|
| 130 |
+
|
| 131 |
+
Known Hugging Face adapters now stamp verified slice metadata automatically:
|
| 132 |
+
|
| 133 |
+
- `gauravgiri/nepali-ocr-dataset`: `nepali`, `real`
|
| 134 |
+
- `prashant0919/nepali-synthetic-ocr-lines`: `nepali`, `synthetic`
|
| 135 |
+
- `nvidia/OCR-Synthetic-Multilingual-v1 --hf-subset en`: `english`, `synthetic`, `line_crop`, `en`
|
| 136 |
+
|
| 137 |
+
Use `rebalance-manifest` only on the training split after `split-manifest`. Rebalancing before the split would turn deliberate duplication into leakage.
|
| 138 |
+
|
| 139 |
+
Use `audit-recognizer-corpus` before launching a recognizer training job. It
|
| 140 |
+
checks the actual train/eval manifests for sample counts, Latin-script coverage,
|
| 141 |
+
Devanagari coverage, mixed-script examples, real/synthetic training coverage,
|
| 142 |
+
and train/eval leakage. This guardrail was added after the failed PaddleOCR
|
| 143 |
+
mixed v4 branch: a new GPU run should not start unless English preservation and
|
| 144 |
+
Nepali coverage are explicit in the corpus.
|
| 145 |
+
|
| 146 |
+
Use `build-recognizer-corpus` to assemble and upsample the training manifest by
|
| 147 |
+
actual text buckets, not by wishful dataset labels. It tracks `latin_only`,
|
| 148 |
+
`devanagari_only`, `mixed_devanagari_latin`, `real`, and `synthetic` coverage,
|
| 149 |
+
then writes a build JSON/Markdown summary next to the output manifest. It does
|
| 150 |
+
not invent missing categories: if a required bucket is absent from the input
|
| 151 |
+
manifests, the command fails so you know to add real source data before using
|
| 152 |
+
GPU time.
|
| 153 |
+
|
| 154 |
+
`train-recognizer --backend paddleocr` writes
|
| 155 |
+
`recognizer-training-summary.json` alongside the generated PaddleOCR config.
|
| 156 |
+
Keep that file with any remote/GPU training bundle and pass it to
|
| 157 |
+
`package-recognizer --training-summary`; it records manifest hashes, slice
|
| 158 |
+
counts, character coverage, label/config hashes, and launch hyperparameters.
|
| 159 |
+
`bundle-recognizer-training` records the bundled summary path and SHA-256 under
|
| 160 |
+
`training_summary`, and packaged recognizer model cards record the same
|
| 161 |
+
provenance when supplied. For real remote/GPU runs, use
|
| 162 |
+
`bundle-recognizer-training --require-corpus-audit`; it runs the recognizer
|
| 163 |
+
corpus audit and refuses to create the bundle if the corpus does not meet the
|
| 164 |
+
configured English, Devanagari, mixed-script, real/synthetic, and leakage gates.
|
| 165 |
+
The gate also checks isolated Latin-only and Devanagari-only samples; a corpus
|
| 166 |
+
made entirely of mixed-script rows is not enough to protect English preservation
|
| 167 |
+
or Nepali-only recognition.
|
| 168 |
+
|
| 169 |
+
Planned source datasets:
|
| 170 |
+
|
| 171 |
+
- `gauravgiri/nepali-ocr-dataset`
|
| 172 |
+
- `prashant0919/nepali-synthetic-ocr-lines`
|
| 173 |
+
- `nvidia/OCR-Synthetic-Multilingual-v1`
|
| 174 |
+
- `cfilt/RoundTripOCR-nepali`
|
| 175 |
+
|
| 176 |
+
The verified Hugging Face adapters are intentionally dataset-specific where the upstream schema is non-standard:
|
| 177 |
+
|
| 178 |
+
- `gauravgiri/nepali-ocr-dataset`: generic `image` + `text`
|
| 179 |
+
- `prashant0919/nepali-synthetic-ocr-lines`: `metadata.jsonl` plus per-line image files, downloaded lazily for the requested rows
|
| 180 |
+
- `cfilt/RoundTripOCR-nepali`: correction pairs via `ocr` and `correct`
|
| 181 |
+
- `nvidia/OCR-Synthetic-Multilingual-v1`: HDF5 shards with `annotations.line_bboxes`; use `--hf-subset en` for English crops
|
| 182 |
+
|
| 183 |
+
Some upstream corpora are dirty. For example, the current `gauravgiri/nepali-ocr-dataset` sample prefix includes null `text` rows, so use `--skip-invalid` when you want a prepared manifest plus an explicit reject log instead of a hard stop.
|
| 184 |
+
|
| 185 |
+
## Baselines
|
| 186 |
+
|
| 187 |
+
The benchmark command supports:
|
| 188 |
+
|
| 189 |
+
- `ours`
|
| 190 |
+
- `tesseract`
|
| 191 |
+
- `stock-paddle`
|
| 192 |
+
- `glm-ocr`
|
| 193 |
+
- `paddleocr-vl`
|
| 194 |
+
|
| 195 |
+
`glm-ocr` and `paddleocr-vl` are external command hooks:
|
| 196 |
+
|
| 197 |
+
```bash
|
| 198 |
+
export OCRTECH_GLM_OCR_CMD='python scripts/run_glm_ocr.py --input {input} --out {out}'
|
| 199 |
+
export OCRTECH_PADDLEOCR_VL_CMD='python scripts/run_paddleocr_vl.py --input {input} --out {out}'
|
| 200 |
+
```
|
| 201 |
+
|
| 202 |
+
The command must include both `{input}` and `{out}` placeholders. It may print Markdown to stdout or write `{out}/document.md`; `ocrtech` now normalizes that Markdown into its own `document.json` schema so external VLM baselines can still be scored for text, table cells, and reading order.
|
| 203 |
+
|
| 204 |
+
The bundled wrappers are thin adapters, not magic:
|
| 205 |
+
|
| 206 |
+
- `scripts/run_glm_ocr.py` uses the official `transformers` path from the `zai-org/GLM-OCR` model card.
|
| 207 |
+
- `scripts/run_paddleocr_vl.py` uses the official `PaddleOCRVL(...).predict(...)` path from the `PaddlePaddle/PaddleOCR-VL-1.6` model card.
|
| 208 |
+
- GLM-OCR still expects a git `transformers` build upstream. On `k2`, the released `transformers 4.57.6` package was not enough to instantiate its processor classes; a dedicated Python 3.11 env with git `transformers` and `torchvision` was required before the wrapper would run.
|
| 209 |
+
- PaddleOCR-VL still needs the OCR extras that the official docs call out, for example `pip install "paddlex[ocr]==3.7.1"`.
|
| 210 |
+
|
| 211 |
+
## Validation Before SOTA Claims
|
| 212 |
+
|
| 213 |
+
Use `validate-claim` before making any quality claim. The default validator is intentionally strict and will fail unless the benchmark report has held-out sample coverage, required slices, leakage checks, paired candidate-vs-baseline gates, and per-gate evidence such as paired wins and optional significance thresholds.
|
| 214 |
+
|
| 215 |
+
Use `recognizer-admission-decision` after a recognizer admission validation run.
|
| 216 |
+
It writes a small JSON/Markdown decision artifact: `admit` only for
|
| 217 |
+
`claim_status=validated` with `passed=true`, otherwise `reject` with failed
|
| 218 |
+
gate metrics and next actions for the next model branch.
|
| 219 |
+
|
| 220 |
+
Use `create-eval-pack` for internal mixed Nepali/English document tests and `audit-eval-pack` to check whether the pack is claim-ready. The internal pack now includes a real `multi_page` slice in addition to the single-page paragraph, table, form, and figure templates. Text packs are for pipeline smoke tests only; image packs are required for OCR claims.
|
| 221 |
+
|
| 222 |
+
Use `preflight-claim` before `run-experiment` on any claim-grade run. It fails early on missing references, invalid model cards, missing baseline dependencies, non-claim-ready eval packs, bad validation configs, and train/eval leakage.
|
| 223 |
+
|
| 224 |
+
For routed or thresholded candidates such as `quality_select_composite` and `script_select_composite`, claim-grade runs also require explicit calibration provenance in the model card. If the candidate was calibrated on the same eval manifest you are about to benchmark, `preflight-claim` now fails on purpose.
|
| 225 |
+
|
| 226 |
+
For those routed candidates, claim-grade runs also need the raw component baselines in the same benchmark. If a `script_select_composite` candidate is built from Tesseract and Surya, the claim run must include `candidate,tesseract,surya`; otherwise the repo now fails preflight because distinctness cannot be checked honestly.
|
| 227 |
+
|
| 228 |
+
Use `run-experiment` to benchmark every manifest input, write reproducibility metadata, and run claim validation in one output directory.
|
| 229 |
+
|
| 230 |
+
Use `review-claim` after the internal pack and public benchmark suites have all been run. It audits the portfolio-level evidence bundle so one green experiment does not get misrepresented as a broad SOTA claim.
|
| 231 |
+
|
| 232 |
+
Use a candidate model card for any real benchmark of `ours`; see [docs/MODELS.md](docs/MODELS.md).
|
| 233 |
+
Use `filter-manifest` when a baseline is too expensive to run on the full pack and you need a reproducible exploratory subset instead of hand-editing a JSONL.
|
| 234 |
+
|
| 235 |
+
See [docs/PREFLIGHT.md](docs/PREFLIGHT.md), [docs/CLAIM_REVIEW.md](docs/CLAIM_REVIEW.md), [docs/EVAL_PACK.md](docs/EVAL_PACK.md), [docs/MODELS.md](docs/MODELS.md), [docs/EXPERIMENTS.md](docs/EXPERIMENTS.md), and [docs/VALIDATION.md](docs/VALIDATION.md).
|
| 236 |
+
|
| 237 |
+
## Remote Training
|
| 238 |
+
|
| 239 |
+
Use `docs/REMOTE_TRAINING.md` for the current `k2` host notes, `audit-remote-host`, and the safe SSH workflow. Do not commit SSH passwords or machine-specific secrets.
|
| 240 |
+
|
| 241 |
+
## Current HF Result
|
| 242 |
+
|
| 243 |
+
The strongest verified TrOCR candidate so far is the rebalanced mixed-script run trained on `k2` and evaluated on a held-out `180`-sample mixed Nepali/English line set.
|
| 244 |
+
|
| 245 |
+
That run is materially better than our own earlier HF candidates:
|
| 246 |
+
|
| 247 |
+
- overall benchmark CER: `1.628157` -> `0.586139`
|
| 248 |
+
- English CER: `0.379729` -> `0.300454`
|
| 249 |
+
- Nepali CER: `7.870295` -> `2.014560`
|
| 250 |
+
- real-slice CER: `12.702701` -> `1.770016`
|
| 251 |
+
|
| 252 |
+
But the first named baseline comparison is still bad news for the project claim: on that same eval split, `tesseract` is better than the rebalanced candidate on overall, English, Nepali, real, and synthetic OCR metrics.
|
| 253 |
+
|
| 254 |
+
The next attempted rescue path also failed. On June 14, 2026, a ByT5 text-correction model trained on `k2` from mined Tesseract noisy/clean pairs was packaged as a `text_correction_composite` candidate with raw Tesseract as the base engine. On the same held-out `180`-sample eval, that composite regressed catastrophically: overall CER `6.912065` versus raw Tesseract `0.329012`, with paired CER `0` wins, `170` losses, and `10` ties. That rules out the current Tesseract-plus-corrector path as a serious answer.
|
| 255 |
+
|
| 256 |
+
There is one narrower improvement worth keeping, but it is not a claim-grade win. A routed Tesseract composite now exists as `quality_select_composite`: it runs default Tesseract (`psm=6`) and a secondary variant (`psm=13`), scores both outputs with the local `quality_score`, and switches only when the secondary beats the primary by a configured margin.
|
| 257 |
+
|
| 258 |
+
The same-pack tuned run looked slightly better on the original `180`-sample mixed eval, but that was not honest enough. After splitting that pack into a separate calibration half (`89` samples) and untouched holdout half (`91` samples), the calibrated routed candidate still improved overall CER on the holdout, just by much less:
|
| 259 |
+
|
| 260 |
+
- raw tesseract holdout overall CER: `0.342234`
|
| 261 |
+
- routed tesseract holdout overall CER: `0.340608`
|
| 262 |
+
- paired CER on holdout: `2` wins, `1` loss, `88` ties
|
| 263 |
+
|
| 264 |
+
That makes it a plausible engineering baseline, not a frontier result. The effect size is tiny, the pack still is not claim-ready, and the real Devanagari recognizer problem remains open.
|
| 265 |
+
|
| 266 |
+
We also tested a richer calibrated Tesseract ensemble over `psm=6/7/11/13` with learned quality-score biases. That looked better on the calibration half, but it failed on the untouched holdout: overall CER regressed from `0.342234` to `0.358352`. That is exactly the kind of result the new calibration workflow is supposed to expose.
|
| 267 |
+
|
| 268 |
+
The more useful June 14, 2026 result is a calibrated Tesseract-plus-Surya script router on `k2`. Raw Surya was not competitive overall on the `91`-sample holdout (`2.198362` CER vs raw Tesseract `0.342234`), but it was materially better on the Nepali slice. A routed `script_select_composite` candidate calibrated on the separate `89`-sample half and evaluated on the untouched `91`-sample holdout did better than raw Tesseract across every reported slice:
|
| 269 |
+
|
| 270 |
+
- raw tesseract holdout overall CER: `0.342234`
|
| 271 |
+
- calibrated script router overall CER: `0.313084`
|
| 272 |
+
- paired CER on holdout: `9` wins, `0` losses, `82` ties
|
| 273 |
+
|
| 274 |
+
That is the strongest verified baseline improvement in the repo so far. It is still not a frontier claim, because it is one internal line-level split and not the full claim bundle, but it is a real out-of-sample gain and a better direction than the current TrOCR branch. The router now also supports `routing_granularity=page` for mixed multi-page documents; the calibrated `k2` result above is still the original document-level calibration, not a new page-level benchmark.
|
| 275 |
+
|
| 276 |
+
We also ran an exploratory page-level script router on the same `91`-sample holdout, reusing the document-level threshold `0.4`. The result was identical to the document-level router (`0.313084` overall CER, `9` wins, `0` losses, `82` ties vs raw Tesseract). That is expected for this pack because it is dominated by single-page line-crop samples; page routing needs a real multi-page document pack before it can prove anything new.
|
| 277 |
+
|
| 278 |
+
The internal image eval pack now has a real rendered `multi_page` slice with bundled page images and actual drawn table grids. On that small rendered smoke pack, the plain script router already beat Tesseract strongly on text (`0.243224` vs `0.540584` overall CER) and on the dedicated table page (`table_cell_f1` `1.0` vs `0.0`). A follow-up exploratory model with `secondary_structure_backfill=true` kept the same CER and backfilled Surya table structure on the multi-page report, lifting aggregate `table_cell_f1` from `0.5` to `1.0` on that smoke pack. That is still not claim-grade evidence because the pack is tiny and only `clean`, but it is a real document-capability improvement.
|
| 279 |
+
|
| 280 |
+
The next step was to stop calling it smoke evidence and run the same backfill router on a claim-ready rendered image pack. That broader internal pack contains `25` samples across `clean`, `scan`, `blur`, `low_contrast`, and `skew`, and its audit passed cleanly. The calibrated backfill router also passed `preflight-claim` on that pack. Against raw Tesseract on the same rendered claim pack it still held up on text and structure:
|
| 281 |
+
|
| 282 |
+
- overall CER: `0.248654` vs `0.689214`
|
| 283 |
+
- blur CER: `0.260039` vs `0.991613`
|
| 284 |
+
- scan CER: `0.237424` vs `0.593631`
|
| 285 |
+
- skew CER: `0.239317` vs `0.766067`
|
| 286 |
+
- aggregate `table_cell_f1`: `0.955556` vs `0.0`
|
| 287 |
+
- aggregate `table_exact_match`: `0.8` vs `0.0`
|
| 288 |
+
- paired CER: `9` wins, `1` loss, `15` ties
|
| 289 |
+
|
| 290 |
+
That result is good enough to keep and bad enough to describe plainly. The multi-page slice tied Tesseract on CER (`0.315789` each) while still recovering table structure that Tesseract missed (`table_cell_f1` `0.911111` vs `0.0`). The form slice regressed slightly on CER (`0.231933` vs `0.230010`). Aggregate reading-order pair accuracy on this rendered pack is also still worse than Tesseract (`0.764` vs `0.816`). So the current candidate is meaningfully better on OCR robustness and table extraction for this internal rendered pack, but it is not yet a clean document-understanding leader.
|
| 291 |
+
|
| 292 |
+
We then widened the comparison beyond Tesseract. Two more June 14, 2026 checks on `k2` matter:
|
| 293 |
+
|
| 294 |
+
- raw Surya on the same `25`-sample rendered claim pack is better on text CER than the calibrated backfill router: `0.195260` vs `0.248654`
|
| 295 |
+
- but the router is still not pointless: it ties Surya on aggregate `table_cell_f1` (`0.955556` each) and is slightly better on aggregate `reading_order_pair_accuracy` (`0.764000` vs `0.750909`)
|
| 296 |
+
|
| 297 |
+
That means the current routed candidate is still not “better than what is out there” on this rendered document pack. Raw Surya is the stronger text OCR baseline here. The router is only defending a narrower position: it preserves comparable structure while staying closer to the existing Tesseract-centered stack.
|
| 298 |
+
|
| 299 |
+
We also fixed a real integration bug in the local Paddle baseline. `paddleocr 3.7.0` on `k2` returns v3 `OCRResult` objects, and the old adapter in this repo treated them like the older tuple/list API and silently dropped their text. After fixing that parser and installing `paddlepaddle 3.3.1`, `stock-paddle` became a real benchmark target on `k2`. On this same rendered pack it is still a weak baseline overall:
|
| 300 |
+
|
| 301 |
+
- stock-Paddle overall CER: `1.749081`
|
| 302 |
+
- stock-Paddle aggregate `table_cell_f1`: `0.0`
|
| 303 |
+
- stock-Paddle aggregate `reading_order_pair_accuracy`: `1.0`
|
| 304 |
+
|
| 305 |
+
So the current picture is sharper now, not prettier: the routed candidate beats Tesseract and this stock-Paddle configuration on OCR robustness and table extraction, but it still loses to raw Surya on text CER.
|
| 306 |
+
|
| 307 |
+
We then used the new `filter-manifest` helper to run a broader but still tractable rendered-document comparison on `k2`: a `5`-sample blur-only subset spanning paragraph, table, form, figure, and multi-page layouts. All `30` runs completed without error across `candidate`, `tesseract`, `surya`, `stock-paddle`, `glm-ocr`, and `paddleocr-vl`. The result sharpened the ranking again:
|
| 308 |
+
|
| 309 |
+
- candidate CER: `0.223273`
|
| 310 |
+
- raw Surya CER: `0.197601`
|
| 311 |
+
- Tesseract CER: `0.732364`
|
| 312 |
+
- stock-Paddle CER: `2.810426`
|
| 313 |
+
- GLM-OCR CER: `2.820269`
|
| 314 |
+
- PaddleOCR-VL CER: `18.470663`
|
| 315 |
+
|
| 316 |
+
On that subset, the current backfill candidate still held the structure line where Tesseract and the weaker VLM baselines collapsed (`table_cell_f1 = 1.0` for the candidate versus `0.0` for Tesseract, stock-Paddle, GLM-OCR, and PaddleOCR-VL). But it still did not beat raw Surya on text CER. That keeps the same honest conclusion intact: on rendered document OCR text, the strongest baseline in this repo is still Surya, not the current routed candidate.
|
| 317 |
+
|
| 318 |
+
The stricter rendered-document result is now explicit too. On `k2`, a larger rendered `claimpack-rendered-v2` split was calibrated separately and evaluated on an untouched `50`-sample holdout. That calibration selected `primary_script_threshold=0.0`, so the routed `script_select_composite` collapsed to "always pick Surya" on the calibration set. On the holdout:
|
| 319 |
+
|
| 320 |
+
- candidate CER: `0.180893`
|
| 321 |
+
- raw Surya CER: `0.180893`
|
| 322 |
+
- candidate `table_cell_f1`: `0.947500`
|
| 323 |
+
- raw Surya `table_cell_f1`: `0.970833`
|
| 324 |
+
|
| 325 |
+
That is the ceiling of the current branch, not a win. The candidate no longer even has a text-vs-structure tradeoff story there; on this larger rendered holdout it effectively degenerates to Surya on text and still loses slightly on table structure. The claim guardrails now reject this case directly: routed candidates that collapse to one engine across all calibration samples fail `candidate-calibration` in both `preflight-claim` and `review-claim`.
|
| 326 |
+
|
| 327 |
+
We also tested whether a quality-based Surya/Tesseract selector could recover anything on that same rendered-v2 setup. It did not. The rendered-v2 Surya quality router calibrated to `selected_threshold=0.0`, then landed at:
|
| 328 |
+
|
| 329 |
+
- candidate CER: `0.181128`
|
| 330 |
+
- raw Surya CER: `0.180893`
|
| 331 |
+
- candidate `table_cell_f1`: `0.947500`
|
| 332 |
+
- raw Surya `table_cell_f1`: `0.947500`
|
| 333 |
+
- paired CER vs Surya: `0` wins, `1` loss, `49` ties
|
| 334 |
+
|
| 335 |
+
That is not a distinct better engine. It is basically Surya with one extra mistake.
|
| 336 |
+
|
| 337 |
+
We then tested `line_align_composite` on the same rendered-v2 holdout: keep Surya text, align it to Tesseract's line scaffold, and see whether the ordering gain is worth the merge. It was not:
|
| 338 |
+
|
| 339 |
+
- candidate CER: `0.310549`
|
| 340 |
+
- raw Surya CER: `0.180893`
|
| 341 |
+
- candidate `table_cell_f1`: `0.897500`
|
| 342 |
+
- raw Surya `table_cell_f1`: `0.970833`
|
| 343 |
+
- candidate `reading_order_pair_accuracy`: `0.792727`
|
| 344 |
+
- raw Surya `reading_order_pair_accuracy`: `0.750909`
|
| 345 |
+
- paired CER vs Surya: `0` wins, `7` losses, `43` ties
|
| 346 |
+
|
| 347 |
+
That is another useful paper result, but not a winning model branch. Better line ordering alone did not rescue the Surya/Tesseract composite family.
|
| 348 |
+
|
| 349 |
+
The upper bound is now measured too. A sample-level oracle that chooses the better of raw Surya and raw Tesseract per holdout sample using ground-truth CER landed at:
|
| 350 |
+
|
| 351 |
+
- oracle CER: `0.180529`
|
| 352 |
+
- raw Surya CER: `0.180893`
|
| 353 |
+
- raw Tesseract CER: `0.714207`
|
| 354 |
+
- oracle choices: Surya `49`, Tesseract `1`
|
| 355 |
+
- paired CER vs Surya: `1` win, `0` losses, `49` ties
|
| 356 |
+
|
| 357 |
+
That closes the routing branch for text OCR. There is almost no remaining sample-level headroom in the Surya/Tesseract component family on rendered-v2. The next credible model-side step has to use a genuinely stronger recognizer or document model, not another selector over these same two engines.
|
| 358 |
+
|
| 359 |
+
See [docs/EXPERIMENTS.md](docs/EXPERIMENTS.md) for the exact commands, artifact paths, and the current evidence gap.
|
| 360 |
+
|
| 361 |
+
If you are turning this work into a paper, start from [docs/PAPER_NOTES.md](docs/PAPER_NOTES.md). Then use [docs/PAPER_RESULTS.md](docs/PAPER_RESULTS.md) for the current generated comparison tables and branch chronology tied to local artifacts.
|
| 362 |
+
|
| 363 |
+
## References
|
| 364 |
+
|
| 365 |
+
- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)
|
| 366 |
+
- [PP-StructureV3 docs](https://paddlepaddle.github.io/PaddleOCR/latest/version3.x/pipeline_usage/PP-StructureV3.html)
|
| 367 |
+
- [PaddleX table structure recognition docs](https://paddlepaddle.github.io/PaddleX/3.4/en/module_usage/tutorials/ocr_modules/table_structure_recognition.html)
|
| 368 |
+
- [PaddleOCR-VL-1.6](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6)
|
| 369 |
+
- [Tesseract docs](https://tesseract-ocr.github.io/docs/)
|
| 370 |
+
- [GLM-OCR](https://huggingface.co/zai-org/GLM-OCR)
|
| 371 |
+
- [Surya OCR](https://github.com/datalab-to/surya)
|
ocr-tech-mps-integration/RESEARCH.md
ADDED
|
@@ -0,0 +1,375 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Research State: Nepal Language Pack OCR
|
| 2 |
+
|
| 3 |
+
Updated: 2026-08-09
|
| 4 |
+
|
| 5 |
+
## Current Goal
|
| 6 |
+
|
| 7 |
+
The end goal is an OCR system covering the multiple languages of the Nepal pack,
|
| 8 |
+
built out of Paddle, **with Limbu and Newar (Nepal Bhasa) as the first two focus
|
| 9 |
+
languages** (user steer, 2026-07-04). The system should accept real book/page images
|
| 10 |
+
or PDFs for any language in the pack and output editable Unicode text with page,
|
| 11 |
+
line, crop, confidence, and provenance evidence preserved. Limbu stays the proven
|
| 12 |
+
template — it has the strongest recognizer and the deepest harvest. Newar is the
|
| 13 |
+
deliberate second: it has its own native script (Prachalit), real cultural demand,
|
| 14 |
+
and is currently the weakest trained recognizer, so it exercises every part of the
|
| 15 |
+
method that Limbu already proved. Every stage that was built Limbu-specific must
|
| 16 |
+
generalize: harvest, silver-label extraction, recognizer training, script routing,
|
| 17 |
+
page/book OCR, and eval. The product is the full pipeline: page
|
| 18 |
+
detection, script-aware routing to per-script recognizers, target-region and
|
| 19 |
+
furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and
|
| 20 |
+
review artifacts for hard lines. YOLO line detection stays a diagnostic/probe;
|
| 21 |
+
Paddle-first is the production story.
|
| 22 |
+
|
| 23 |
+
## The Language Pack
|
| 24 |
+
|
| 25 |
+
Two inventories exist and they are not the same thing:
|
| 26 |
+
|
| 27 |
+
- The registry `languages/nepal-v1.json` tracks 18 target languages (17 Nepal + English)
|
| 28 |
+
with scripts, fonts, dictionaries, and corpus status.
|
| 29 |
+
- The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found
|
| 30 |
+
**50 normalized languages (53 raw labels) in 400 articles over ~7 months** (see the
|
| 31 |
+
harvest section). Labels include all registry languages plus Chamling Rai, Bahing,
|
| 32 |
+
Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo,
|
| 33 |
+
Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi,
|
| 34 |
+
Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more.
|
| 35 |
+
Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा,
|
| 36 |
+
ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) —
|
| 37 |
+
`tools/build_gorkhapatra_language_pack_inventory.py` owns that normalization and the
|
| 38 |
+
label→registry mapping.
|
| 39 |
+
|
| 40 |
+
Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded
|
| 41 |
+
2026-06-18 scorecard in `docs/RECOGNIZER_GOAL_PROGRESS.md`):
|
| 42 |
+
|
| 43 |
+
- **Limbu/Sirijonga 0.041 real-print** (champion:
|
| 44 |
+
`runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23`,
|
| 45 |
+
cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval).
|
| 46 |
+
- **8 Devanagari languages 0.075–0.129 real-print** (Majhi, Magar, Sunuwar-Deva, Magahi,
|
| 47 |
+
Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer.
|
| 48 |
+
- **Sunuwar (Jenticha) 0.121 real-crop** after the same-font retrain (0.342 synth-only).
|
| 49 |
+
- **Tirhuta 0.343 and Newa 0.613 synthetic-only** — weak, transliteration-fed, honest
|
| 50 |
+
fails; Newa needs real printed-register Nepal Lipi, not more pixels.
|
| 51 |
+
- **Kirat Rai (70 real crops) and Lepcha (132 real crops)** blocked on legacy-font
|
| 52 |
+
converter coverage outside their eval editions; **Gurung Khema** has no real print
|
| 53 |
+
anywhere (revival script) — synthetic-only by nature, claim-scoped accordingly.
|
| 54 |
+
|
| 55 |
+
The near-term pack claim should therefore be script-family shaped: one Sirijonga
|
| 56 |
+
recognizer, one shared Devanagari recognizer covering the many Devanagari-written
|
| 57 |
+
languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa,
|
| 58 |
+
Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval
|
| 59 |
+
crops exist.
|
| 60 |
+
|
| 61 |
+
**Focus order (2026-07-04 steer): Limbu first, Newar second.** Newar state: the
|
| 62 |
+
Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own
|
| 63 |
+
synthetic eval; transliteration-fed training failed; real printed-register Nepal
|
| 64 |
+
Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar
|
| 65 |
+
route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20
|
| 66 |
+
Gorkhapatra pages; `newar` added to the registry (Devanagari primary + Prachalit
|
| 67 |
+
secondary; audit passes); a frozen by-date pack built at
|
| 68 |
+
`outputs/newar-devanagari-silver-pack-v1/` (train 4,096 / dev 1,528 / test 1,139
|
| 69 |
+
lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently
|
| 70 |
+
verified; pack sha256
|
| 71 |
+
`1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85`); and the
|
| 72 |
+
**first real-print Devanagari-Newar baseline** was scored locally (CPU export
|
| 73 |
+
route, `tools/score_recognizer_pack.py`, per `docs/LOCAL_RECOGNIZER_SCORING.md`):
|
| 74 |
+
the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at
|
| 75 |
+
**codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158** (n=1,139 silver
|
| 76 |
+
lines — machine-converted labels, diagnostic, not claim-grade). The same-day
|
| 77 |
+
warm-start fine-tune (**newar-ft-v1**: deva-v2 base, LR 2e-5 cosine, 60 epochs,
|
| 78 |
+
batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance
|
| 79 |
+
`ashish-experiments`, ~68 min; artifacts + train log at
|
| 80 |
+
`outputs/aws-newar-deva-ft-v1/`) scores the same frozen test at **codepoint CER
|
| 81 |
+
0.0465, grapheme CER 0.0955, exact-line 0.245** — a 25/27% relative CER drop and
|
| 82 |
+
+55% exact-line over the deva-v2 baseline, scored through the identical local
|
| 83 |
+
CPU path. **Newar's grapheme CER now sits inside deva-v2's own-language range
|
| 84 |
+
(0.075–0.129): the second focus language reached Devanagari-pack parity from
|
| 85 |
+
zero in one day.** Same claim boundary: silver machine-converted labels,
|
| 86 |
+
diagnostic, not claim-grade until native review. Next Newar lever: native
|
| 87 |
+
Prachalit (real printed-register data), and page-level Newar eval for the
|
| 88 |
+
region-filtering track.
|
| 89 |
+
|
| 90 |
+
## Source Rule
|
| 91 |
+
|
| 92 |
+
For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata
|
| 93 |
+
only. Use `https://gorkhapatraonline.com/publications/naya-nepal` to find the language,
|
| 94 |
+
date, and article, but do not use the blog/card image for OCR, training, or evaluation
|
| 95 |
+
when the full paper exists. The authoritative source is the matching Gorkhapatra epaper
|
| 96 |
+
PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus
|
| 97 |
+
the text layer extracted from that same PDF page, with PDF URL/hash, page number,
|
| 98 |
+
rendered image hash, extraction command/version, font evidence, and converter/map hash
|
| 99 |
+
recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with
|
| 100 |
+
`src/ocrtech/limbu_legacy.py` + `data/mappings/limbu-legacy/Limbu.map`. The same
|
| 101 |
+
pattern — classify the text layer, then build an explicit converter only when the font
|
| 102 |
+
requires it — replicates to every other language. Manual transcription is the fallback
|
| 103 |
+
for no usable text layer, not the default.
|
| 104 |
+
|
| 105 |
+
## Getting More Gorkhapatra Pages (the specified harvest recipe)
|
| 106 |
+
|
| 107 |
+
All numbers below were verified live on 2026-07-04.
|
| 108 |
+
|
| 109 |
+
**Archive size.** The Naya Nepal publication archive paginates at
|
| 110 |
+
`https://gorkhapatraonline.com/publications/naya-nepal?page=N` with 5 articles per
|
| 111 |
+
page and runs to page ~352 before drying up: **~1,750 language-page articles total**
|
| 112 |
+
(about 2 per day going back years). The 60-page crawl already yielded 300 items across
|
| 113 |
+
53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly
|
| 114 |
+
older dates.
|
| 115 |
+
|
| 116 |
+
**Epaper endpoint behavior (the critical constraint).** The date-range endpoint
|
| 117 |
+
`https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra`
|
| 118 |
+
**caps results at 30 PDFs per query and ignores `page=N` pagination**. A wide window
|
| 119 |
+
silently drops most dates — the 2026-07-04 six-window discovery pass aligned only
|
| 120 |
+
111/300 items for exactly this reason, with unmatched dates interleaved among matched
|
| 121 |
+
ones. **Seven-day windows return complete daily coverage (7/7 verified).** Always
|
| 122 |
+
harvest with weekly windows.
|
| 123 |
+
|
| 124 |
+
**The recipe** (one command; add one `--epaper-url` per week to cover the publication
|
| 125 |
+
date span being harvested):
|
| 126 |
+
|
| 127 |
+
```bash
|
| 128 |
+
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
|
| 129 |
+
ocrtech audit-gorkhapatra-language-pages \
|
| 130 |
+
--out outputs/gorkhapatra-language-pages-full-<date> \
|
| 131 |
+
--publication-url https://gorkhapatraonline.com/publications/naya-nepal \
|
| 132 |
+
--publication-pages 80 \
|
| 133 |
+
--epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \
|
| 134 |
+
--epaper-url '...one per week, going back...' \
|
| 135 |
+
--download-pdfs \
|
| 136 |
+
--render-pages \
|
| 137 |
+
--timeout-seconds 180
|
| 138 |
+
```
|
| 139 |
+
|
| 140 |
+
The command aligns every publication card (all languages at once — alignment is
|
| 141 |
+
language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL),
|
| 142 |
+
searches pages for the language-page markers (`भाषा पृष्ठ` and its legacy-encoding
|
| 143 |
+
form `efiff k[i7`), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly
|
| 144 |
+
windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with
|
| 145 |
+
**400 publication items, 195 aligned dates, 394 rendered language-page hits, zero
|
| 146 |
+
warnings** at `outputs/gorkhapatra-language-pages-full-2026-07-04/`. Batch 2
|
| 147 |
+
(2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more
|
| 148 |
+
items: **798 items, 294 aligned dates, 589 further page hits**, at
|
| 149 |
+
`outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/`. The merged normalized
|
| 150 |
+
inventory (`tools/build_gorkhapatra_language_pack_inventory.py`) shows **56
|
| 151 |
+
languages, 983 rendered language-page hits, most languages with ~20–27 pages
|
| 152 |
+
spanning 14 months**, at `outputs/language-pack-inventory-merged-2026-07-04/`. For
|
| 153 |
+
comparison, the entire previous state of the project had 8 usable pages for one
|
| 154 |
+
language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least
|
| 155 |
+
March 2024 — verified the epaper endpoint serves complete weeks there) is another
|
| 156 |
+
~2× and stays month-batchable with the same recipe.
|
| 157 |
+
|
| 158 |
+
**Scaling to the full archive.** Full depth is ~700+ unique dates ≈ 35 GB of PDFs.
|
| 159 |
+
Do not do it in one run: batch by month (4–5 weekly windows + the matching
|
| 160 |
+
publication-page depth), run `extract-gorkhapatra-language-page-pdf-text` per batch,
|
| 161 |
+
then keep rendered pages, extracted text, and hashes; PDFs can be pruned after
|
| 162 |
+
extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume
|
| 163 |
+
(`cdjk@100.110.148.43:/Volumes/TRANSCEND`) is the right home for a full-depth mirror.
|
| 164 |
+
|
| 165 |
+
**Per-language downstream.** After harvest, run per language:
|
| 166 |
+
|
| 167 |
+
```bash
|
| 168 |
+
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
|
| 169 |
+
ocrtech extract-gorkhapatra-language-page-pdf-text \
|
| 170 |
+
<pack-manifest.jsonl> \
|
| 171 |
+
--candidate-language <label> \
|
| 172 |
+
--target-font <font> \
|
| 173 |
+
--out outputs/<language>-pdf-text-<date>
|
| 174 |
+
```
|
| 175 |
+
|
| 176 |
+
then classify each page's text layer (native Unicode / legacy font / Devanagari
|
| 177 |
+
transcription / Latin / none) and only then decide converter vs manual transcription.
|
| 178 |
+
|
| 179 |
+
**Font/text-layer facts already established** (30-page, 15-language extraction at
|
| 180 |
+
`outputs/gorkhapatra-multilingual-language-pages-current-2026-06-24-pdf-text/`):
|
| 181 |
+
|
| 182 |
+
- The body font of non-Limbu language pages is **`nayanepal` — a legacy 8-bit
|
| 183 |
+
Preeti-family encoding, not Unicode** (8,845 of 11,791 spans; the text layer reads
|
| 184 |
+
`k[i7 ;+of]hs` where the page shows पृष्ठ संयोजक). **The converter already
|
| 185 |
+
exists**: `nepal_ttf2utf.convert_devanagari(font="nayanepal")` in the sibling repo
|
| 186 |
+
`/Users/cdjk/github/llm/nepal-ttf2utf` (Preeti base map + nayanepal extension
|
| 187 |
+
glyphs + NFC, ~97–99% clean on real Gorkhapatra pages), already used by
|
| 188 |
+
`tools/build_gorkhapatra_devanagari_real.py` for the 79k-row real Devanagari
|
| 189 |
+
corpus. The remaining leverage is wiring it **per language**: run it over each
|
| 190 |
+
harvested language page and emit per-language silver line packs with the same
|
| 191 |
+
faithfulness gate (clean-Unicode allow-set, `conversion_gap` accounting, leakage
|
| 192 |
+
checks) that the Devanagari pack builder already implements. The same sibling repo
|
| 193 |
+
also hosts the limbu, kiratrai, lepcha, and sunuwar legacy converters.
|
| 194 |
+
- Tibetan-script fonts (`TibetanMachine`, `MonlamUniOuChan5`, `Qomolangma-*`) appear on
|
| 195 |
+
Mugal and related pages, but **every such span has empty extracted text (verified
|
| 196 |
+
2026-07-04 across all 2026-06-24 samples)** — Tibetan-script pages are visual-only,
|
| 197 |
+
like the known Limbu `2082-11-12` date. They need OCR bootstrap or manual
|
| 198 |
+
transcription; no converter is possible.
|
| 199 |
+
- `Gorkhapatra-*`, `Nagarik-*`, `TimesNewRoman*`, `MyriadPro-*` are masthead/furniture
|
| 200 |
+
fonts — useful as non-target filters for target-region selection.
|
| 201 |
+
- `NamdhinggoSILL` appears only on Limbu dates (the existing Limbu path).
|
| 202 |
+
|
| 203 |
+
## What We Built (Limbu template, still true)
|
| 204 |
+
|
| 205 |
+
The Gorkhapatra alignment path: publication-page discovery, epaper-date matching, PDF
|
| 206 |
+
download/rendering, page-marker search, PDF-native span/font extraction, review
|
| 207 |
+
sheets, reference templates, and dashboard visualization. The expanded Limbu harvest
|
| 208 |
+
has 9 discovered dates, 8 usable PDF-text-layer pages, 2,441 line rows, 2,430 usable
|
| 209 |
+
silver labels (`outputs/limbu-gorkhapatra-pdf-backed-silver-2026-07-03-v1/`).
|
| 210 |
+
|
| 211 |
+
The Limbu recognizer training/eval path: 48,507-row combined pack; real-heavy
|
| 212 |
+
fine-tune champion at codepoint CER 0.011336 / grapheme 0.024736 / exact-line 0.729 on
|
| 213 |
+
the frozen 255-line silver eval. Synthetic hardening lessons: targeted+Unicode packs
|
| 214 |
+
regressed; v3 dictionary warm-start catastrophically failed (~0.98 CER); keep the
|
| 215 |
+
real-heavy v2 champion; future gains come from real/book-domain harvesting or a
|
| 216 |
+
deliberate rehead, not blind synth rows.
|
| 217 |
+
|
| 218 |
+
The page/book OCR loop: local + HF Space (`spaces/limbu-ocr/`, deployed private at
|
| 219 |
+
`https://huggingface.co/spaces/ampixa/limbu-ocr`) with line OCR, Paddle Page OCR,
|
| 220 |
+
Paddle Book OCR, image-line fallback, runtime checks, annotated pages, and crops.
|
| 221 |
+
|
| 222 |
+
Page-level diagnostics on the 8-page silver pack: Rows LTR + y/conf70 filter scores
|
| 223 |
+
page cp CER 0.414330, line F1 0.963009, matched-line cp CER 0.026039, order accuracy
|
| 224 |
+
0.993176. Recognition on matched lines is strong; **full-page CER is dominated by page
|
| 225 |
+
reconstruction, target-region selection, and furniture filtering** — that is the
|
| 226 |
+
pipeline problem to solve, and it is language-independent.
|
| 227 |
+
|
| 228 |
+
Supervised line-detector infrastructure from CDC PDF geometry (287 pages, 26,160
|
| 229 |
+
annotations, mAP50 ≈ 0.921 on PDF-rendered val) — evidence the detector path works,
|
| 230 |
+
not a solved product.
|
| 231 |
+
|
| 232 |
+
A claim-grade eval harness (versioned packs, hashes, provenance, normalization
|
| 233 |
+
profiles, cp/grapheme CER, CIs, paired comparisons, leakage manifests) — parked as a
|
| 234 |
+
guardrail; do not let it eat work blocks that should go to the pipeline.
|
| 235 |
+
|
| 236 |
+
## Current Working State
|
| 237 |
+
|
| 238 |
+
The strongest component is the Sirijonga line recognizer. The weakest component is
|
| 239 |
+
end-to-end page/book reconstruction: target-region selection, furniture filtering,
|
| 240 |
+
line preservation, and ordering. Of the three Limbu-locked code spots, two were
|
| 241 |
+
generalized on 2026-07-04 (Codex worker, tests green): `_script_ratio` in
|
| 242 |
+
`src/ocrtech/engines.py` is now driven by the shared `SCRIPT_CODEPOINT_RANGES` table
|
| 243 |
+
covering all pack scripts (devanagari, limbu/sirijonga, latin, sunuwar/jenticha,
|
| 244 |
+
lepcha, kirat_rai, newa/prachalit, tirhuta, gurung_khema, ol_chiki, tibetan, with
|
| 245 |
+
aliases), and the image-line filter in `src/ocrtech/image_lines.py` accepts any
|
| 246 |
+
script key from that same table. The third spot fell the same day: the
|
| 247 |
+
`src/ocrtech/limbu_runtime.py` preflight now takes `required_script` (default
|
| 248 |
+
`limbu`, validated against the shared table; CLI `--required-script`), so a router
|
| 249 |
+
card for any pack script can be preflighted. The engine classes themselves (`CandidateEngine`, `PaddleOcrEngine`,
|
| 250 |
+
`ScriptSelectCompositeEngine`, `build_image_line_document`, `structure.py` reading
|
| 251 |
+
order) are already parameterized — the rest of the generalization is registry-driven
|
| 252 |
+
wiring, not a rewrite.
|
| 253 |
+
|
| 254 |
+
## Apple MPS Recognizer Training Evidence (2026-08-09)
|
| 255 |
+
|
| 256 |
+
Stock PaddlePaddle is not the immediate Apple-GPU route. A direct PaddlePaddle
|
| 257 |
+
3.3.1 probe on `k2` exposed no custom devices and rejected `mps`, `metal`, and
|
| 258 |
+
`custom_device:metal`. Paddle's official `PaddleCustomDevice` MPS backend exists,
|
| 259 |
+
but is currently a version-0.0.1 prototype with too small an operator surface to
|
| 260 |
+
establish PP-OCRv5 training.
|
| 261 |
+
|
| 262 |
+
The practical path is an Apache-2.0 vendored subset of
|
| 263 |
+
`frotms/PaddleOCR2Pytorch` pinned at
|
| 264 |
+
`c05307fbd61e575d55f0fef0334026c7536aec76`: PPLCNetV3 scale 0.95 plus the
|
| 265 |
+
SVTR/CTC branch of the PP-OCRv5 mobile recognizer. Dynamic pooling preserves the
|
| 266 |
+
recognition width (`T = input_width / 4`), and a localized contiguity fix before
|
| 267 |
+
the SVTR encoder's `conv3` makes the model safe for the tested MPS runtimes. The
|
| 268 |
+
model and log-softmax run on MPS; `CTCLoss` runs on CPU and autograd returns
|
| 269 |
+
finite gradients to MPS for the optimizer update. The NRTR auxiliary branch is
|
| 270 |
+
frozen and not part of this CTC loss, so this is not a complete reproduction of
|
| 271 |
+
PaddleOCR's multi-loss training recipe.
|
| 272 |
+
|
| 273 |
+
One-step evidence is green on both Macs. The end-to-end `mbp2` run used Python
|
| 274 |
+
3.11.15 and Torch 2.13.0: `[4,3,48,160] -> [4,40,7]`, loss 29.883989, 582 finite
|
| 275 |
+
MPS gradient tensors, and a non-zero optimizer update across synthetic Latin,
|
| 276 |
+
Devanagari, and Limbu samples. Direct width-320 forward/backward/update checks
|
| 277 |
+
also passed on `mbp2` and on `k2` with Torch 2.12. These cold one-step timings
|
| 278 |
+
are diagnostics, not benchmarks.
|
| 279 |
+
|
| 280 |
+
The checkpoint boundary is strict: this backend writes PyTorch checkpoints,
|
| 281 |
+
not Paddle `.pdparams`; conversion requires explicit tensor mapping and frozen
|
| 282 |
+
inference-parity tests. `mbp2` and `k2` should run independent experiments with
|
| 283 |
+
distinct seeds and run directories, not multi-host MPS DDP. The remaining proof
|
| 284 |
+
is a multi-epoch real-manifest run with resume/reload, memory observation, and
|
| 285 |
+
canonical held-out CER. See `docs/PADDLEOCR_MPS.md` for exact smoke artifacts,
|
| 286 |
+
hashes, compatibility limits, and host guidance.
|
| 287 |
+
|
| 288 |
+
## The Paddle Pipeline (whole pipeline, not just recognizers)
|
| 289 |
+
|
| 290 |
+
Target architecture, all stages Paddle-first and language-pack-parameterized by the
|
| 291 |
+
per-language pipeline config **`configs/nepal-language-pack-pipeline-v1.json`**
|
| 292 |
+
(50 languages: label, registry id, scripts, recognizer artifact, dictionary,
|
| 293 |
+
converter, text-layer class, harvest counts — generated 2026-07-04 from the
|
| 294 |
+
inventory):
|
| 295 |
+
|
| 296 |
+
1. **Capture/prep** — existing `prepare-limbu-capture` path, already
|
| 297 |
+
language-agnostic in behavior; rename/rescope.
|
| 298 |
+
2. **Detection** — Paddle text detection as production default; YOLO line detector and
|
| 299 |
+
morphology image-line boxes stay as diagnostics/rescue passes.
|
| 300 |
+
3. **Script identification & routing** — generalize `_script_ratio` to a
|
| 301 |
+
Unicode-block-driven scorer over the pack's script list (Devanagari, Limbu,
|
| 302 |
+
Jenticha/Sunuwar, Lepcha, Kirat Rai, Newa, Tirhuta, Gurung Khema, Ol Chiki,
|
| 303 |
+
Tibetan, Latin), then route each line to the script's recognizer from the registry.
|
| 304 |
+
4. **Recognition** — per-script PP-OCRv5 recognizers (existing runs) with per-script
|
| 305 |
+
dictionaries and input-width profiles (Sirijonga width-480 lesson).
|
| 306 |
+
5. **Target-region & furniture filtering** — non-oracle region selection using layout
|
| 307 |
+
plus font/language signals; furniture fonts and masthead geometry as negative
|
| 308 |
+
evidence. This is the main open research item; score against the Rows LTR +
|
| 309 |
+
y/conf70 baseline.
|
| 310 |
+
6. **Reading order & assembly** — `auto_layout` selection (wide-column RTL vs rows)
|
| 311 |
+
per page, conservative assembly stays diagnostic.
|
| 312 |
+
7. **Export** — Unicode text, JSON with provenance, Markdown; review queues +
|
| 313 |
+
dashboards per page (existing Limbu artifact shapes, renamed pack-wide).
|
| 314 |
+
8. **Eval guardrail** — frozen real-page packs per language: page CER/WER, line
|
| 315 |
+
P/R/F1, reading order, blank/extra/missed lines, review burden.
|
| 316 |
+
|
| 317 |
+
PP-StructureV3 and PaddleOCR-VL remain the documented fallback/teacher direction for
|
| 318 |
+
hard layouts, per `docs/NEPAL_MULTILINGUAL_OCR_GOAL.md`.
|
| 319 |
+
|
| 320 |
+
## Next Work
|
| 321 |
+
|
| 322 |
+
1. ~~Finish the 2026-07-04 full harvest and inventory~~ **Done 2026-07-04**: 394
|
| 323 |
+
language-page hits, 50 languages, inventory manifests at
|
| 324 |
+
`outputs/language-pack-inventory-full-2026-07-04/`.
|
| 325 |
+
2. ~~Per-language silver packs via the existing `nayanepal` converter~~ **Done
|
| 326 |
+
2026-07-04** (`tools/build_language_pack_silver_line_packs.py`, tests green): the
|
| 327 |
+
batch-1 run produced **116,287 kept silver lines across 50 languages, zero
|
| 328 |
+
errors**, with per-line provenance (PDF hash, page, bbox, raw legacy bytes,
|
| 329 |
+
converted text, image hash) at `outputs/language-pack-silver-lines-2026-07-04/`;
|
| 330 |
+
spot-checked conversions read as clean, language-plausible Devanagari. The
|
| 331 |
+
batch-2 run over the older 589 page hits kept **183,324 more lines (49 languages,
|
| 332 |
+
zero errors)** at `outputs/language-pack-silver-lines-batch2-2026-07-04/` —
|
| 333 |
+
**299,611 per-language-attributed silver lines total** across the two batches
|
| 334 |
+
(~2.4 GB of crops). Run the tool with
|
| 335 |
+
`uv run --python 3.11 --with pymupdf --with npttf2utf` — `nepal_ttf2utf` needs the
|
| 336 |
+
`npttf2utf` base-map package at runtime, and without it every span fails
|
| 337 |
+
(cleanly, with per-span error records, but kept=0). **Honest caveat**: lines
|
| 338 |
+
are page-attributed, not language-verified — a language page also carries Nepali
|
| 339 |
+
furniture/boxes, and pages whose body is not nayanepal Devanagari (Urdu, the
|
| 340 |
+
Tibetan-script pages) yield mostly furniture lines. A language-ID/lexicon filter
|
| 341 |
+
before per-language recognizer training is the follow-up gate.
|
| 342 |
+
3. **Newar track (second focus language)**: add the Newar registry entry
|
| 343 |
+
(Devanagari + Prachalit rows); build the Devanagari-written Newar recognizer
|
| 344 |
+
eval/train split from the 6,800 harvested silver lines (by-date split, frozen
|
| 345 |
+
test); keep native Prachalit as the separate longer arc — real printed-register
|
| 346 |
+
Nepal Lipi harvesting, not more transliteration, is the lever.
|
| 347 |
+
4. **Classify Tibetan-script pages** (Hyolmo/Mugal/Bhote) and decide converter vs
|
| 348 |
+
native Unicode.
|
| 349 |
+
5. ~~Generalize the pipeline wiring~~ **Done 2026-07-04**: all three Limbu-locked
|
| 350 |
+
spots generalized (shared script table, filter keys, `--required-script`
|
| 351 |
+
preflight), tests green; `configs/nepal-language-pack-pipeline-v1.json` holds the
|
| 352 |
+
per-language wiring.
|
| 353 |
+
6. **Target-region/furniture filtering stage** for Paddle page/book OCR, scored
|
| 354 |
+
against the current baseline on the 8-page Limbu pack and at least one
|
| 355 |
+
Devanagari-language pack (Newar is the natural candidate given the focus order).
|
| 356 |
+
7. Only after page reconstruction improves, export high-quality real/silver line
|
| 357 |
+
crops back into recognizer training, per language.
|
| 358 |
+
8. Month-batch the archive back-harvest toward the full ~1,750-article depth, storing
|
| 359 |
+
the mirror on the remote 2 TB volume.
|
| 360 |
+
|
| 361 |
+
## Mistakes To Avoid
|
| 362 |
+
|
| 363 |
+
Do not use Naya Nepal blog images as training/eval evidence when the PDF exists. Do
|
| 364 |
+
not use wide epaper date windows — the endpoint silently caps at 30 PDFs per query;
|
| 365 |
+
weekly windows only. Do not treat unnormalized Gorkhapatra labels as distinct
|
| 366 |
+
languages (थारु vs थारू). Do not treat the `nayanepal` text layer as Unicode — it is
|
| 367 |
+
legacy Preeti-family and needs the converter before any silver label is real. Do not
|
| 368 |
+
report Paddle `norm_edit_dis` as CER; real CER comes from `tools/eval_metrics.py`
|
| 369 |
+
micro CER over canonical predictions. Do not treat synthetic data as claim evidence,
|
| 370 |
+
and label Gurung Khema results as synthetic-by-nature. Do not promote the
|
| 371 |
+
targeted+Unicode checkpoint or v3 dictionary warm-starts. Do not spend long blocks on
|
| 372 |
+
generic eval-harness work unless it directly improves the book/page OCR loop. Do not
|
| 373 |
+
frame video-to-page work as OCR scope; reduce video to stills first. Do not present
|
| 374 |
+
page OCR as solved: matched-line recognition is strong, but full-page CER is still
|
| 375 |
+
dominated by reconstruction and region selection.
|
ocr-tech-mps-integration/configs/cdjk-language-corpus-migration-v1.json
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"id": "cdjk-language-corpus-migration-v1",
|
| 3 |
+
"created_date": "2026-06-16",
|
| 4 |
+
"source_host": "cdjk@100.110.148.43",
|
| 5 |
+
"source_machine": "cdjks-MacBook-Pro.local",
|
| 6 |
+
"source_storage": {
|
| 7 |
+
"primary_external_volume": "/Volumes/TRANSCEND",
|
| 8 |
+
"primary_external_volume_size": "3.6Ti",
|
| 9 |
+
"primary_external_volume_available_at_inventory": "2.0Ti"
|
| 10 |
+
},
|
| 11 |
+
"destination_root": "data/external-language-resources/cdjk-100.110.148.43",
|
| 12 |
+
"copy_policy": {
|
| 13 |
+
"purpose": "migrate text corpora, dictionaries, lexicons, grammar notes, and manifests for OCR synthesis planning",
|
| 14 |
+
"include_suffixes": [".txt", ".tsv", ".csv", ".json", ".jsonl", ".md", ".yaml", ".yml"],
|
| 15 |
+
"excluded_trees": [".git", ".pytest_cache", ".playwright-mcp", ".ruff_cache", "__pycache__", "node_modules", ".venv", ".venvs"],
|
| 16 |
+
"excluded_suffixes": [
|
| 17 |
+
".wav",
|
| 18 |
+
".mp3",
|
| 19 |
+
".flac",
|
| 20 |
+
".m4a",
|
| 21 |
+
".mp4",
|
| 22 |
+
".mov",
|
| 23 |
+
".png",
|
| 24 |
+
".jpg",
|
| 25 |
+
".jpeg",
|
| 26 |
+
".pdf",
|
| 27 |
+
".zip",
|
| 28 |
+
".pt",
|
| 29 |
+
".pth",
|
| 30 |
+
".onnx",
|
| 31 |
+
".safetensors",
|
| 32 |
+
".bin",
|
| 33 |
+
".ckpt"
|
| 34 |
+
],
|
| 35 |
+
"notes": [
|
| 36 |
+
"The migration is intentionally selective because the local machine had about 33GiB free at copy time.",
|
| 37 |
+
"Generated data under data/ is ignored by git; this config records the reproducible migration policy."
|
| 38 |
+
]
|
| 39 |
+
},
|
| 40 |
+
"source_roots": [
|
| 41 |
+
{
|
| 42 |
+
"label": "dhimal-toolkit",
|
| 43 |
+
"remote_path": "/Users/cdjk/github/llm/dhimal-toolkit",
|
| 44 |
+
"local_path": "data/external-language-resources/cdjk-100.110.148.43/dhimal-toolkit"
|
| 45 |
+
},
|
| 46 |
+
{
|
| 47 |
+
"label": "gurung-language-toolkit",
|
| 48 |
+
"remote_path": "/Users/cdjk/github/llm/gurung-language-toolkit",
|
| 49 |
+
"local_path": "data/external-language-resources/cdjk-100.110.148.43/gurung-language-toolkit"
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"label": "limbu-speech-toolkit",
|
| 53 |
+
"remote_path": "/Users/cdjk/github/llm/limbu-speech-toolkit",
|
| 54 |
+
"local_path": "data/external-language-resources/cdjk-100.110.148.43/limbu-speech-toolkit"
|
| 55 |
+
},
|
| 56 |
+
{
|
| 57 |
+
"label": "ltk",
|
| 58 |
+
"remote_path": "/Users/cdjk/github/llm/ltk",
|
| 59 |
+
"local_path": "data/external-language-resources/cdjk-100.110.148.43/ltk"
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"label": "magar-toolkit",
|
| 63 |
+
"remote_path": "/Users/cdjk/github/llm/magar-toolkit",
|
| 64 |
+
"local_path": "data/external-language-resources/cdjk-100.110.148.43/magar-toolkit"
|
| 65 |
+
},
|
| 66 |
+
{
|
| 67 |
+
"label": "rai-speech-toolkit",
|
| 68 |
+
"remote_path": "/Users/cdjk/github/llm/rai-speech-toolkit",
|
| 69 |
+
"local_path": "data/external-language-resources/cdjk-100.110.148.43/rai-speech-toolkit"
|
| 70 |
+
}
|
| 71 |
+
],
|
| 72 |
+
"migration_result": {
|
| 73 |
+
"transferred_files": 2307,
|
| 74 |
+
"total_files_after_copy": 2308,
|
| 75 |
+
"total_size_bytes": 1689175338,
|
| 76 |
+
"local_size_human": "1.6G",
|
| 77 |
+
"suffix_counts": {
|
| 78 |
+
"json": 1071,
|
| 79 |
+
"txt": 686,
|
| 80 |
+
"md": 209,
|
| 81 |
+
"jsonl": 179,
|
| 82 |
+
"tsv": 155,
|
| 83 |
+
"csv": 4,
|
| 84 |
+
"yml": 3
|
| 85 |
+
}
|
| 86 |
+
},
|
| 87 |
+
"audits": {
|
| 88 |
+
"resource_audit": "outputs/cdjk-language-resource-audit-2026-06-16/synthesis-resource-audit.json",
|
| 89 |
+
"limbu_text_audit": "outputs/cdjk-limbu-limdic-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 90 |
+
"limbu_only_text_audit": "outputs/cdjk-limbu-only-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 91 |
+
"tamang_text_audit": "outputs/cdjk-tamang-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 92 |
+
"magar_text_audit": "outputs/cdjk-magar-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 93 |
+
"gurung_text_audit": "outputs/cdjk-gurung-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 94 |
+
"dhimal_parallel_text_audit": "outputs/cdjk-dhimal-parallel-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 95 |
+
"dhimal_mono_text_audit": "outputs/cdjk-dhimal-mono-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 96 |
+
"magar_parallel_extra_text_audit": "outputs/cdjk-magar-parallel-extra-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 97 |
+
"magar_mono_extra_text_audit": "outputs/cdjk-magar-mono-extra-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 98 |
+
"rai_bantawa_text_audit": "outputs/cdjk-rai-bantawa-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 99 |
+
"rai_chamling_text_audit": "outputs/cdjk-rai-chamling-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 100 |
+
"rai_kulung_text_audit": "outputs/cdjk-rai-kulung-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 101 |
+
"rai_athpariya_text_audit": "outputs/cdjk-rai-athpariya-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 102 |
+
"rai_khaling_text_audit": "outputs/cdjk-rai-khaling-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 103 |
+
"rai_yakkha_text_audit": "outputs/cdjk-rai-yakkha-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 104 |
+
"rai_dumi_text_audit": "outputs/cdjk-rai-dumi-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 105 |
+
"rai_thulung_text_audit": "outputs/cdjk-rai-thulung-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 106 |
+
"rai_koyu_text_audit": "outputs/cdjk-rai-koyu-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 107 |
+
"rai_lohorung_text_audit": "outputs/cdjk-rai-lohorung-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 108 |
+
"rai_sunwar_text_audit": "outputs/cdjk-rai-sunwar-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 109 |
+
"rai_limbu_text_audit": "outputs/cdjk-rai-limbu-text-audit-2026-06-16/synthesis-text-manifest-audit.json",
|
| 110 |
+
"limbu_rendered_smoke_audit": "outputs/cdjk-limbu-rendered-smoke-audit-2026-06-16-limbu-only/rendered-line-manifest-audit.json"
|
| 111 |
+
},
|
| 112 |
+
"prepared_outputs": {
|
| 113 |
+
"limbu_limbu_only_text": "data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered/limbu-only.jsonl",
|
| 114 |
+
"limbu_all_limdic_text": "data/synthesis-text/cdjk-limbu-limdic-2026-06-16/limbu-limdic-text.jsonl",
|
| 115 |
+
"tamang_text": "data/synthesis-text/cdjk-tamang-2026-06-16/tamang-text.jsonl",
|
| 116 |
+
"magar_text": "data/synthesis-text/cdjk-magar-2026-06-16/magar-text.jsonl",
|
| 117 |
+
"gurung_text": "data/synthesis-text/cdjk-gurung-2026-06-16/toolkit-parallel-text.jsonl",
|
| 118 |
+
"dhimal_parallel_text": "data/synthesis-text/cdjk-dhimal-parallel-2026-06-16/toolkit-parallel-text.jsonl",
|
| 119 |
+
"dhimal_mono_text": "data/synthesis-text/cdjk-dhimal-mono-2026-06-16/toolkit-parallel-text.jsonl",
|
| 120 |
+
"magar_parallel_extra_text": "data/synthesis-text/cdjk-magar-parallel-extra-2026-06-16/toolkit-parallel-text.jsonl",
|
| 121 |
+
"magar_mono_extra_text": "data/synthesis-text/cdjk-magar-mono-extra-2026-06-16/toolkit-parallel-text.jsonl",
|
| 122 |
+
"rai_bantawa_text": "data/synthesis-text/cdjk-rai-bantawa-2026-06-16/toolkit-parallel-text.jsonl",
|
| 123 |
+
"rai_chamling_text": "data/synthesis-text/cdjk-rai-chamling-2026-06-16/toolkit-parallel-text.jsonl",
|
| 124 |
+
"rai_kulung_text": "data/synthesis-text/cdjk-rai-kulung-2026-06-16/toolkit-parallel-text.jsonl",
|
| 125 |
+
"rai_athpariya_text": "data/synthesis-text/cdjk-rai-athpariya-2026-06-16/toolkit-parallel-text.jsonl",
|
| 126 |
+
"rai_khaling_text": "data/synthesis-text/cdjk-rai-khaling-2026-06-16/toolkit-parallel-text.jsonl",
|
| 127 |
+
"rai_yakkha_text": "data/synthesis-text/cdjk-rai-yakkha-2026-06-16/toolkit-parallel-text.jsonl",
|
| 128 |
+
"rai_dumi_text": "data/synthesis-text/cdjk-rai-dumi-2026-06-16/toolkit-parallel-text.jsonl",
|
| 129 |
+
"rai_thulung_text": "data/synthesis-text/cdjk-rai-thulung-2026-06-16/toolkit-parallel-text.jsonl",
|
| 130 |
+
"rai_koyu_text": "data/synthesis-text/cdjk-rai-koyu-2026-06-16/toolkit-parallel-text.jsonl",
|
| 131 |
+
"rai_lohorung_text": "data/synthesis-text/cdjk-rai-lohorung-2026-06-16/toolkit-parallel-text.jsonl",
|
| 132 |
+
"rai_sunwar_text": "data/synthesis-text/cdjk-rai-sunwar-2026-06-16/toolkit-parallel-text.jsonl",
|
| 133 |
+
"rai_limbu_text": "data/synthesis-text/cdjk-rai-limbu-2026-06-16/toolkit-parallel-text.jsonl",
|
| 134 |
+
"limbu_rendered_smoke_manifest": "data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only/manifests/rendered-line-manifest.jsonl",
|
| 135 |
+
"limbu_rendered_smoke_labels": "data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only/labels/train.txt"
|
| 136 |
+
},
|
| 137 |
+
"prepared_manifest_summary": {
|
| 138 |
+
"manifest_count": 20,
|
| 139 |
+
"total_samples": 228212,
|
| 140 |
+
"total_duplicates": 45258,
|
| 141 |
+
"script_counts": {
|
| 142 |
+
"Devanagari": 202560,
|
| 143 |
+
"Mixed_Devanagari_Latin": 10337,
|
| 144 |
+
"Latin": 6521,
|
| 145 |
+
"Limbu": 8772,
|
| 146 |
+
"Mixed_Limbu_Devanagari": 14,
|
| 147 |
+
"Mixed_Limbu_Latin": 3,
|
| 148 |
+
"Other": 5
|
| 149 |
+
}
|
| 150 |
+
},
|
| 151 |
+
"language_coverage_leads": [
|
| 152 |
+
"limbu",
|
| 153 |
+
"tamang",
|
| 154 |
+
"magar",
|
| 155 |
+
"gurung",
|
| 156 |
+
"dhimal",
|
| 157 |
+
"bantawa",
|
| 158 |
+
"chamling",
|
| 159 |
+
"athpariya",
|
| 160 |
+
"khaling",
|
| 161 |
+
"kulung",
|
| 162 |
+
"sampang",
|
| 163 |
+
"sunwar",
|
| 164 |
+
"thulung",
|
| 165 |
+
"lohorung",
|
| 166 |
+
"yakkha",
|
| 167 |
+
"dumi",
|
| 168 |
+
"koyu"
|
| 169 |
+
]
|
| 170 |
+
}
|
ocr-tech-mps-integration/configs/claim-review-sota-v1.json
ADDED
|
@@ -0,0 +1,121 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"require_shared_candidate_model": true,
|
| 3 |
+
"require_fallback_model_present": true,
|
| 4 |
+
"require_candidate_fallback_provenance": true,
|
| 5 |
+
"require_shared_fallback_model": true,
|
| 6 |
+
"require_model_audit_passed": true,
|
| 7 |
+
"require_fallback_model_audit_passed": true,
|
| 8 |
+
"require_trained_recognizer": true,
|
| 9 |
+
"require_model_admission": true,
|
| 10 |
+
"required_experiments": [
|
| 11 |
+
{
|
| 12 |
+
"name": "internal-nepali-english-document-pack",
|
| 13 |
+
"datasets": ["ocrtech-eval-pack"],
|
| 14 |
+
"require_validation_status": "validated",
|
| 15 |
+
"required_validation_config": "configs/validation-internal-sota-v1.json",
|
| 16 |
+
"require_baselines": ["candidate", "surya", "tesseract", "stock-paddle", "glm-ocr", "paddleocr-vl"],
|
| 17 |
+
"required_metrics": [
|
| 18 |
+
"cer",
|
| 19 |
+
"wer",
|
| 20 |
+
"document_reconstruction_similarity",
|
| 21 |
+
"table_cell_f1",
|
| 22 |
+
"reading_order_pair_accuracy",
|
| 23 |
+
"figure_detection_f1",
|
| 24 |
+
"figure_caption_cer"
|
| 25 |
+
],
|
| 26 |
+
"required_metric_slices": {
|
| 27 |
+
"cer": ["nepali", "english"],
|
| 28 |
+
"wer": ["nepali", "english"],
|
| 29 |
+
"document_reconstruction_similarity": ["table", "reading_order", "figure"],
|
| 30 |
+
"table_cell_f1": ["table"],
|
| 31 |
+
"reading_order_pair_accuracy": ["reading_order"],
|
| 32 |
+
"figure_detection_f1": ["figure"],
|
| 33 |
+
"figure_caption_cer": ["figure"]
|
| 34 |
+
},
|
| 35 |
+
"required_output_artifacts": ["document_json", "document_md", "quality_json"],
|
| 36 |
+
"required_output_artifact_slices": {
|
| 37 |
+
"table_csv_html": ["table"],
|
| 38 |
+
"figure_metadata": ["figure"],
|
| 39 |
+
"figure_images": ["figure"]
|
| 40 |
+
},
|
| 41 |
+
"required_validation_gates": [
|
| 42 |
+
{"baseline": "tesseract", "metric": "cer", "slices": ["nepali"]},
|
| 43 |
+
{"baseline": "tesseract", "metric": "wer", "slices": ["nepali"]},
|
| 44 |
+
{"baseline": "stock-paddle", "metric": "cer", "slices": ["nepali"]},
|
| 45 |
+
{"baseline": "stock-paddle", "metric": "cer", "slices": ["english"]},
|
| 46 |
+
{"baseline": "stock-paddle", "metric": "wer", "slices": ["english"]},
|
| 47 |
+
{"baseline": "glm-ocr", "metric": "cer", "slices": ["nepali"]},
|
| 48 |
+
{"baseline": "surya", "metric": "table_cell_f1", "slices": ["table"]},
|
| 49 |
+
{"baseline": "surya", "metric": "figure_detection_f1", "slices": ["figure"]},
|
| 50 |
+
{"baseline": "surya", "metric": "figure_caption_cer", "slices": ["figure"]},
|
| 51 |
+
{"baseline": "paddleocr-vl", "metric": "reading_order_pair_accuracy", "slices": ["reading_order"]},
|
| 52 |
+
{"baseline": "paddleocr-vl", "metric": "document_reconstruction_similarity", "slices": ["figure"]}
|
| 53 |
+
],
|
| 54 |
+
"require_candidate_fallback_metrics": true,
|
| 55 |
+
"max_candidate_fallback_trigger_rate": 0.35,
|
| 56 |
+
"max_candidate_fallback_trigger_rate_by_slice": {
|
| 57 |
+
"english": 0.10,
|
| 58 |
+
"nepali": 0.20
|
| 59 |
+
},
|
| 60 |
+
"max_candidate_fallback_failed_rate": 0.0,
|
| 61 |
+
"required_slices": ["nepali", "english", "table", "reading_order", "figure"]
|
| 62 |
+
},
|
| 63 |
+
{
|
| 64 |
+
"name": "public-omnidocbench",
|
| 65 |
+
"datasets": ["opendatalab/OmniDocBench"],
|
| 66 |
+
"require_validation_status": "validated",
|
| 67 |
+
"required_validation_config": "configs/validation-public-omnidocbench-sota-v1.json",
|
| 68 |
+
"require_baselines": ["candidate", "surya", "paddleocr-vl"],
|
| 69 |
+
"required_metrics": ["cer", "document_reconstruction_similarity", "table_cell_f1", "table_structure_similarity", "reading_order_pair_accuracy"],
|
| 70 |
+
"required_metric_slices": {
|
| 71 |
+
"table_cell_f1": ["document_parsing"],
|
| 72 |
+
"table_structure_similarity": ["document_parsing"],
|
| 73 |
+
"reading_order_pair_accuracy": ["document_parsing"],
|
| 74 |
+
"document_reconstruction_similarity": ["document_parsing"]
|
| 75 |
+
},
|
| 76 |
+
"required_output_artifacts": ["document_json", "document_md", "quality_json"],
|
| 77 |
+
"required_output_artifact_slices": {
|
| 78 |
+
"table_csv_html": ["table"]
|
| 79 |
+
},
|
| 80 |
+
"required_validation_gates": [
|
| 81 |
+
{"baseline": "surya", "metric": "cer", "slices": ["public", "document_parsing"]},
|
| 82 |
+
{"baseline": "surya", "metric": "table_cell_f1", "slices": ["public", "table"]},
|
| 83 |
+
{"baseline": "paddleocr-vl", "metric": "document_reconstruction_similarity", "slices": ["public", "document_parsing"]}
|
| 84 |
+
],
|
| 85 |
+
"require_candidate_fallback_metrics": true,
|
| 86 |
+
"max_candidate_fallback_trigger_rate": 0.35,
|
| 87 |
+
"max_candidate_fallback_trigger_rate_by_slice": {
|
| 88 |
+
"public": 0.35
|
| 89 |
+
},
|
| 90 |
+
"max_candidate_fallback_failed_rate": 0.0,
|
| 91 |
+
"required_slices": ["public", "document_parsing"]
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"name": "public-real5-omnidocbench-or-olmocr",
|
| 95 |
+
"eval_manifest_patterns": ["*Real5-OmniDocBench*", "*olmOCR-bench*"],
|
| 96 |
+
"require_validation_status": "validated",
|
| 97 |
+
"required_validation_config": "configs/validation-public-real5-sota-v1.json",
|
| 98 |
+
"require_baselines": ["candidate", "surya", "glm-ocr", "paddleocr-vl"],
|
| 99 |
+
"required_metrics": ["cer", "wer", "document_reconstruction_similarity", "reading_order_pair_accuracy"],
|
| 100 |
+
"required_metric_slices": {
|
| 101 |
+
"document_reconstruction_similarity": ["public"],
|
| 102 |
+
"reading_order_pair_accuracy": ["public"]
|
| 103 |
+
},
|
| 104 |
+
"required_output_artifacts": ["document_json", "document_md", "quality_json"],
|
| 105 |
+
"required_validation_gates": [
|
| 106 |
+
{"baseline": "surya", "metric": "cer", "slices": ["public"]},
|
| 107 |
+
{"baseline": "surya", "metric": "wer", "slices": ["public"]},
|
| 108 |
+
{"baseline": "glm-ocr", "metric": "cer", "slices": ["public"]},
|
| 109 |
+
{"baseline": "glm-ocr", "metric": "wer", "slices": ["public"]},
|
| 110 |
+
{"baseline": "paddleocr-vl", "metric": "document_reconstruction_similarity", "slices": ["public"]}
|
| 111 |
+
],
|
| 112 |
+
"require_candidate_fallback_metrics": true,
|
| 113 |
+
"max_candidate_fallback_trigger_rate": 0.35,
|
| 114 |
+
"max_candidate_fallback_trigger_rate_by_slice": {
|
| 115 |
+
"public": 0.35
|
| 116 |
+
},
|
| 117 |
+
"max_candidate_fallback_failed_rate": 0.0,
|
| 118 |
+
"required_slices": ["public"]
|
| 119 |
+
}
|
| 120 |
+
]
|
| 121 |
+
}
|
ocr-tech-mps-integration/configs/eval-harness-recipes-v1.json
ADDED
|
@@ -0,0 +1,651 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"version": "eval-harness-recipes-v1",
|
| 3 |
+
"updated": "2026-06-19",
|
| 4 |
+
"recipes": [
|
| 5 |
+
{
|
| 6 |
+
"id": "newa-zenodo-native-retrain-oracle-crop",
|
| 7 |
+
"task": "recognizer_oracle_crop",
|
| 8 |
+
"language": "newa",
|
| 9 |
+
"script": "Newa",
|
| 10 |
+
"model_family": "newa-native-paddleocr-remote",
|
| 11 |
+
"runner": {
|
| 12 |
+
"type": "external_command",
|
| 13 |
+
"command_template": "REPLACE_WITH_REMOTE_NEWA_PREDICT_COMMAND --labels {label_file} --data {data_dir} --out {prediction_file}",
|
| 14 |
+
"requires_operator_resolution": true,
|
| 15 |
+
"notes": "GPU checkpoint path is not checked into this repo; resolve from the remote native-retrain handoff before scoring."
|
| 16 |
+
},
|
| 17 |
+
"eval_pack": {
|
| 18 |
+
"label_file": "outputs/eval-packs/newa-zenodo-6967421-v1/eval.txt",
|
| 19 |
+
"data_dir": "outputs/eval-packs/newa-zenodo-6967421-v1",
|
| 20 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 21 |
+
"manifest_file": "outputs/eval-packs/newa-zenodo-6967421-v1/manifests/recognizer-eval.jsonl",
|
| 22 |
+
"summary_file": "outputs/eval-packs/newa-zenodo-6967421-v1/summary.json",
|
| 23 |
+
"dataset": "newa-zenodo-6967421-prachalit-real",
|
| 24 |
+
"domain": "real_handwritten",
|
| 25 |
+
"gt_status": "machine_converted",
|
| 26 |
+
"split": "eval",
|
| 27 |
+
"eval_pack_version": "newa-zenodo-6967421-v1",
|
| 28 |
+
"frozen_eval_pack": true,
|
| 29 |
+
"unseen_for_selection": false,
|
| 30 |
+
"claim_eligible": false
|
| 31 |
+
},
|
| 32 |
+
"leakage_sources": [
|
| 33 |
+
{
|
| 34 |
+
"label_file": "data/newa-native-render/train.txt",
|
| 35 |
+
"data_dir": "data/newa-native-render",
|
| 36 |
+
"split": "train"
|
| 37 |
+
},
|
| 38 |
+
{
|
| 39 |
+
"label_file": "data/newa-native-render/eval.txt",
|
| 40 |
+
"data_dir": "data/newa-native-render",
|
| 41 |
+
"split": "dev"
|
| 42 |
+
}
|
| 43 |
+
],
|
| 44 |
+
"claim_blockers": [
|
| 45 |
+
"canonical eval pack audit is not claim-ready; see outputs/eval-packs/newa-zenodo-6967421-v1/audit/recognizer-eval-pack-audit.json",
|
| 46 |
+
"eval pack not declared unseen for model/checkpoint selection",
|
| 47 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 48 |
+
"row-level claim eligibility is false",
|
| 49 |
+
"remote checkpoint path must be resolved before reproducible scoring"
|
| 50 |
+
]
|
| 51 |
+
},
|
| 52 |
+
{
|
| 53 |
+
"id": "sunuwar-jenticha-samefont-oracle-crop",
|
| 54 |
+
"task": "recognizer_oracle_crop",
|
| 55 |
+
"language": "sunuwar",
|
| 56 |
+
"script": "Sunuwar",
|
| 57 |
+
"model_family": "sunuwar-samefont-paddleocr-remote",
|
| 58 |
+
"runner": {
|
| 59 |
+
"type": "external_command",
|
| 60 |
+
"command_template": "REPLACE_WITH_REMOTE_SUNUWAR_PREDICT_COMMAND --labels {label_file} --data {data_dir} --out {prediction_file}",
|
| 61 |
+
"requires_operator_resolution": true,
|
| 62 |
+
"notes": "Remote same-font checkpoint is referenced by handoff but not packaged locally."
|
| 63 |
+
},
|
| 64 |
+
"eval_pack": {
|
| 65 |
+
"label_file": "outputs/eval-packs/sunuwar-herald-real-v1/eval.txt",
|
| 66 |
+
"data_dir": "outputs/eval-packs/sunuwar-herald-real-v1",
|
| 67 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 68 |
+
"manifest_file": "outputs/eval-packs/sunuwar-herald-real-v1/manifests/recognizer-eval.jsonl",
|
| 69 |
+
"summary_file": "outputs/eval-packs/sunuwar-herald-real-v1/summary.json",
|
| 70 |
+
"dataset": "sikkim-herald-sunuwar-real",
|
| 71 |
+
"domain": "real_print",
|
| 72 |
+
"gt_status": "machine_converted",
|
| 73 |
+
"split": "eval",
|
| 74 |
+
"eval_pack_version": "sunuwar-herald-real-v1",
|
| 75 |
+
"frozen_eval_pack": true,
|
| 76 |
+
"unseen_for_selection": true,
|
| 77 |
+
"claim_eligible": false
|
| 78 |
+
},
|
| 79 |
+
"leakage_sources": [
|
| 80 |
+
{
|
| 81 |
+
"label_file": "data/sunuwar-train-pack/train.txt",
|
| 82 |
+
"data_dir": "data/sunuwar-train-pack",
|
| 83 |
+
"split": "train"
|
| 84 |
+
},
|
| 85 |
+
{
|
| 86 |
+
"label_file": "data/sunuwar-train-pack/eval.txt",
|
| 87 |
+
"data_dir": "data/sunuwar-train-pack",
|
| 88 |
+
"split": "dev"
|
| 89 |
+
}
|
| 90 |
+
],
|
| 91 |
+
"claim_blockers": [
|
| 92 |
+
"canonical eval pack audit is not claim-ready; see outputs/eval-packs/sunuwar-herald-real-v1/audit/recognizer-eval-pack-audit.json",
|
| 93 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 94 |
+
"row-level claim eligibility is false",
|
| 95 |
+
"remote checkpoint path must be resolved before reproducible scoring"
|
| 96 |
+
]
|
| 97 |
+
},
|
| 98 |
+
{
|
| 99 |
+
"id": "lepcha-herald-real-oracle-crop",
|
| 100 |
+
"task": "recognizer_oracle_crop",
|
| 101 |
+
"language": "lepcha",
|
| 102 |
+
"script": "Lepcha",
|
| 103 |
+
"model_family": "lepcha-paddleocr-unresolved",
|
| 104 |
+
"runner": {
|
| 105 |
+
"type": "external_command",
|
| 106 |
+
"command_template": "REPLACE_WITH_LEPCHA_PADDLEOCR_PREDICT_COMMAND --labels {label_file} --data {data_dir} --out {prediction_file}",
|
| 107 |
+
"requires_operator_resolution": true,
|
| 108 |
+
"notes": "No packaged local Lepcha checkpoint is registered yet; resolve the exact recognizer and prediction command before scoring."
|
| 109 |
+
},
|
| 110 |
+
"eval_pack": {
|
| 111 |
+
"label_file": "outputs/eval-packs/lepcha-herald-real-v1/eval.txt",
|
| 112 |
+
"data_dir": "outputs/eval-packs/lepcha-herald-real-v1",
|
| 113 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 114 |
+
"manifest_file": "outputs/eval-packs/lepcha-herald-real-v1/manifests/recognizer-eval.jsonl",
|
| 115 |
+
"summary_file": "outputs/eval-packs/lepcha-herald-real-v1/summary.json",
|
| 116 |
+
"dataset": "sikkim-herald-lepcha-real",
|
| 117 |
+
"domain": "real_print",
|
| 118 |
+
"gt_status": "machine_converted",
|
| 119 |
+
"split": "eval",
|
| 120 |
+
"eval_pack_version": "lepcha-herald-real-v1",
|
| 121 |
+
"frozen_eval_pack": true,
|
| 122 |
+
"unseen_for_selection": true,
|
| 123 |
+
"claim_eligible": false
|
| 124 |
+
},
|
| 125 |
+
"leakage_sources": [
|
| 126 |
+
{
|
| 127 |
+
"label_file": "data/synth-corpora/lepcha/train.txt",
|
| 128 |
+
"data_dir": "data/synth-corpora/lepcha",
|
| 129 |
+
"split": "train"
|
| 130 |
+
},
|
| 131 |
+
{
|
| 132 |
+
"label_file": "data/synth-corpora/lepcha/eval.txt",
|
| 133 |
+
"data_dir": "data/synth-corpora/lepcha",
|
| 134 |
+
"split": "dev"
|
| 135 |
+
}
|
| 136 |
+
],
|
| 137 |
+
"claim_blockers": [
|
| 138 |
+
"canonical eval pack audit is not claim-ready; see outputs/eval-packs/lepcha-herald-real-v1/audit/recognizer-eval-pack-audit.json",
|
| 139 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 140 |
+
"row-level claim eligibility is false",
|
| 141 |
+
"packaged local model card is absent",
|
| 142 |
+
"external prediction command must be resolved before reproducible scoring"
|
| 143 |
+
]
|
| 144 |
+
},
|
| 145 |
+
{
|
| 146 |
+
"id": "kiratrai-herald-real-oracle-crop",
|
| 147 |
+
"task": "recognizer_oracle_crop",
|
| 148 |
+
"language": "kiratrai",
|
| 149 |
+
"script": "KiratRai",
|
| 150 |
+
"model_family": "kiratrai-paddleocr-unresolved",
|
| 151 |
+
"runner": {
|
| 152 |
+
"type": "external_command",
|
| 153 |
+
"command_template": "REPLACE_WITH_KIRATRAI_PADDLEOCR_PREDICT_COMMAND --labels {label_file} --data {data_dir} --out {prediction_file}",
|
| 154 |
+
"requires_operator_resolution": true,
|
| 155 |
+
"notes": "No packaged local Kirat Rai checkpoint is registered yet; resolve the exact recognizer and prediction command before scoring."
|
| 156 |
+
},
|
| 157 |
+
"eval_pack": {
|
| 158 |
+
"label_file": "outputs/eval-packs/kiratrai-herald-real-v1/eval.txt",
|
| 159 |
+
"data_dir": "outputs/eval-packs/kiratrai-herald-real-v1",
|
| 160 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 161 |
+
"summary_file": "outputs/eval-packs/kiratrai-herald-real-v1/summary.json",
|
| 162 |
+
"dataset": "sikkim-herald-kiratrai-real",
|
| 163 |
+
"domain": "real_print",
|
| 164 |
+
"gt_status": "machine_converted",
|
| 165 |
+
"split": "eval",
|
| 166 |
+
"eval_pack_version": "kiratrai-herald-real-v1",
|
| 167 |
+
"frozen_eval_pack": true,
|
| 168 |
+
"unseen_for_selection": true,
|
| 169 |
+
"claim_eligible": false,
|
| 170 |
+
"manifest_file": "outputs/eval-packs/kiratrai-herald-real-v1/manifests/recognizer-eval.jsonl"
|
| 171 |
+
},
|
| 172 |
+
"leakage_sources": [
|
| 173 |
+
{
|
| 174 |
+
"label_file": "data/kiratrai-train/real-renders/labels.txt",
|
| 175 |
+
"data_dir": "data/kiratrai-train/real-renders",
|
| 176 |
+
"split": "train"
|
| 177 |
+
},
|
| 178 |
+
{
|
| 179 |
+
"label_file": "data/synth-corpora/kiratrai/train.txt",
|
| 180 |
+
"data_dir": "data/synth-corpora/kiratrai",
|
| 181 |
+
"split": "train"
|
| 182 |
+
},
|
| 183 |
+
{
|
| 184 |
+
"label_file": "data/synth-corpora/kiratrai/eval.txt",
|
| 185 |
+
"data_dir": "data/synth-corpora/kiratrai",
|
| 186 |
+
"split": "dev"
|
| 187 |
+
}
|
| 188 |
+
],
|
| 189 |
+
"claim_blockers": [
|
| 190 |
+
"canonical eval pack audit is not claim-ready; see outputs/eval-packs/kiratrai-herald-real-v1/audit/recognizer-eval-pack-audit.json",
|
| 191 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 192 |
+
"row-level claim eligibility is false",
|
| 193 |
+
"packaged local model card is absent",
|
| 194 |
+
"external prediction command must be resolved before reproducible scoring"
|
| 195 |
+
]
|
| 196 |
+
},
|
| 197 |
+
{
|
| 198 |
+
"id": "limbu-sirijonga-gorkhapatra-oracle-crop",
|
| 199 |
+
"task": "recognizer_oracle_crop",
|
| 200 |
+
"language": "limbu",
|
| 201 |
+
"script": "Limbu",
|
| 202 |
+
"model_family": "limbu-sirijonga-ppocrv5-mobile",
|
| 203 |
+
"runner": {
|
| 204 |
+
"type": "external_command",
|
| 205 |
+
"command_template": "REPLACE_WITH_LIMBU_PADDLEOCR_PREDICT_COMMAND --labels {label_file} --data {data_dir} --out {prediction_file}",
|
| 206 |
+
"requires_operator_resolution": true,
|
| 207 |
+
"training_summary": "runs/limbu-sirijonga-ppocrv5-mobile-2026-06-16/recognizer-training-summary.json"
|
| 208 |
+
},
|
| 209 |
+
"eval_pack": {
|
| 210 |
+
"label_file": "outputs/eval-packs/limbu-gorkhapatra-real-v1/eval.txt",
|
| 211 |
+
"data_dir": "outputs/eval-packs/limbu-gorkhapatra-real-v1",
|
| 212 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 213 |
+
"dataset": "gorkhapatra-limbu-real-crops",
|
| 214 |
+
"domain": "real_print",
|
| 215 |
+
"gt_status": "machine_converted",
|
| 216 |
+
"split": "eval",
|
| 217 |
+
"eval_pack_version": "limbu-gorkhapatra-real-v1",
|
| 218 |
+
"frozen_eval_pack": true,
|
| 219 |
+
"unseen_for_selection": false,
|
| 220 |
+
"claim_eligible": false,
|
| 221 |
+
"manifest_file": "outputs/eval-packs/limbu-gorkhapatra-real-v1/manifests/recognizer-eval.jsonl",
|
| 222 |
+
"summary_file": "outputs/eval-packs/limbu-gorkhapatra-real-v1/summary.json"
|
| 223 |
+
},
|
| 224 |
+
"leakage_sources": [
|
| 225 |
+
{
|
| 226 |
+
"label_file": "runs/limbu-sirijonga-ppocrv5-mobile-2026-06-16/labels/train.txt",
|
| 227 |
+
"data_dir": ".",
|
| 228 |
+
"split": "train"
|
| 229 |
+
},
|
| 230 |
+
{
|
| 231 |
+
"label_file": "runs/limbu-sirijonga-ppocrv5-mobile-2026-06-16/labels/eval.txt",
|
| 232 |
+
"data_dir": ".",
|
| 233 |
+
"split": "dev"
|
| 234 |
+
}
|
| 235 |
+
],
|
| 236 |
+
"claim_blockers": [
|
| 237 |
+
"canonical eval pack audit is not claim-ready; see outputs/eval-packs/limbu-gorkhapatra-real-v1/audit/recognizer-eval-pack-audit.json",
|
| 238 |
+
"eval pack not declared unseen for model/checkpoint selection",
|
| 239 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 240 |
+
"row-level claim eligibility is false",
|
| 241 |
+
"packaged local model card is absent"
|
| 242 |
+
]
|
| 243 |
+
},
|
| 244 |
+
{
|
| 245 |
+
"id": "limbu-devanagari-cdc-ppocrv5-devclean-heldout",
|
| 246 |
+
"task": "recognizer_oracle_crop",
|
| 247 |
+
"language": "limbu",
|
| 248 |
+
"script": "Devanagari",
|
| 249 |
+
"model_family": "limbu-devanagari-cdc-ppocrv5-mobile",
|
| 250 |
+
"runner": {
|
| 251 |
+
"type": "external_command",
|
| 252 |
+
"command_template": "REPLACE_WITH_LIMBU_DEVANAGARI_PADDLEOCR_PREDICT_COMMAND --labels {label_file} --data {data_dir} --out {prediction_file}",
|
| 253 |
+
"requires_operator_resolution": true,
|
| 254 |
+
"training_summary": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/training-recipes/ppocrv5-mobile-devclean/recognizer-training-summary.json",
|
| 255 |
+
"config": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/training-recipes/ppocrv5-mobile-devclean/paddle_recognizer.yml",
|
| 256 |
+
"dictionary": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/training-recipes/ppocrv5-mobile-devclean/nep_eng_dict.txt",
|
| 257 |
+
"training_bundle": "runs/limbu-devanagari-cdc-ppocrv5-mobile-training-bundle-2026-06-21/limbu-devanagari-cdc-ppocrv5-mobile-devclean-training-bundle.tar.gz",
|
| 258 |
+
"selected_checkpoint_on_test": false,
|
| 259 |
+
"notes": "Train from the dev-clean bundle, select/tune only on the 258-row dev split, then fill in checkpoint and prediction command for this sealed 31-row eval split."
|
| 260 |
+
},
|
| 261 |
+
"eval_pack": {
|
| 262 |
+
"label_file": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/eval.txt",
|
| 263 |
+
"data_dir": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21",
|
| 264 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 265 |
+
"manifest_file": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/manifests/recognizer-eval.jsonl",
|
| 266 |
+
"summary_file": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/summaries/eval.summary.json",
|
| 267 |
+
"script_inventory_file": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/script-inventory.json",
|
| 268 |
+
"dataset": "limbu-devanagari-cdc-line-crop-corpus-draft",
|
| 269 |
+
"domain": "real_print",
|
| 270 |
+
"gt_status": "curriculum_text_unreviewed",
|
| 271 |
+
"split": "eval",
|
| 272 |
+
"eval_pack_version": "limbu-devanagari-cdc-line-crop-corpus-draft-v1",
|
| 273 |
+
"frozen_eval_pack": false,
|
| 274 |
+
"unseen_for_selection": true,
|
| 275 |
+
"claim_eligible": false
|
| 276 |
+
},
|
| 277 |
+
"leakage_sources": [
|
| 278 |
+
{
|
| 279 |
+
"manifest_file": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/train-manifest.jsonl",
|
| 280 |
+
"split": "train_dev"
|
| 281 |
+
}
|
| 282 |
+
],
|
| 283 |
+
"claim_blockers": [
|
| 284 |
+
"trained checkpoint path and reproducible prediction command are unresolved until the PP-OCRv5 bundle is run",
|
| 285 |
+
"eval pack is not frozen",
|
| 286 |
+
"GT is unreviewed curriculum text and lacks human/native-reader calibration",
|
| 287 |
+
"row-level claim eligibility is false",
|
| 288 |
+
"packaged local model card is absent"
|
| 289 |
+
]
|
| 290 |
+
},
|
| 291 |
+
{
|
| 292 |
+
"id": "devanagari-shared-gorkhapatra-oracle-crop",
|
| 293 |
+
"task": "recognizer_oracle_crop",
|
| 294 |
+
"language": "devanagari_shared",
|
| 295 |
+
"script": "Devanagari",
|
| 296 |
+
"model_family": "devanagari-ppocrv5-mobile",
|
| 297 |
+
"runner": {
|
| 298 |
+
"type": "external_command",
|
| 299 |
+
"command_template": "REPLACE_WITH_DEVANAGARI_PADDLEOCR_PREDICT_COMMAND --labels {label_file} --data {data_dir} --out {prediction_file}",
|
| 300 |
+
"requires_operator_resolution": true,
|
| 301 |
+
"training_summary": "runs/devanagari-ppocrv5-mobile-2026-06-17/recognizer-training-summary.json"
|
| 302 |
+
},
|
| 303 |
+
"eval_pack": {
|
| 304 |
+
"label_file": "outputs/eval-packs/gorkhapatra-devanagari-real-v1/eval.txt",
|
| 305 |
+
"data_dir": "outputs/eval-packs/gorkhapatra-devanagari-real-v1",
|
| 306 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 307 |
+
"summary_file": "outputs/eval-packs/gorkhapatra-devanagari-real-v1/summary.json",
|
| 308 |
+
"dataset": "gorkhapatra-devanagari-real",
|
| 309 |
+
"domain": "real_print",
|
| 310 |
+
"gt_status": "machine_converted",
|
| 311 |
+
"split": "eval",
|
| 312 |
+
"eval_pack_version": "gorkhapatra-devanagari-real-v1",
|
| 313 |
+
"frozen_eval_pack": true,
|
| 314 |
+
"unseen_for_selection": false,
|
| 315 |
+
"claim_eligible": false,
|
| 316 |
+
"manifest_file": "outputs/eval-packs/gorkhapatra-devanagari-real-v1/manifests/recognizer-eval.jsonl"
|
| 317 |
+
},
|
| 318 |
+
"leakage_sources": [
|
| 319 |
+
{
|
| 320 |
+
"label_file": "runs/devanagari-ppocrv5-mobile-2026-06-17/labels/train.txt",
|
| 321 |
+
"data_dir": ".",
|
| 322 |
+
"split": "train"
|
| 323 |
+
},
|
| 324 |
+
{
|
| 325 |
+
"label_file": "runs/devanagari-ppocrv5-mobile-2026-06-17/labels/eval.txt",
|
| 326 |
+
"data_dir": ".",
|
| 327 |
+
"split": "dev"
|
| 328 |
+
}
|
| 329 |
+
],
|
| 330 |
+
"claim_blockers": [
|
| 331 |
+
"canonical eval pack audit is not claim-ready; see outputs/eval-packs/gorkhapatra-devanagari-real-v1/audit/recognizer-eval-pack-audit.json",
|
| 332 |
+
"eval pack not declared unseen for model/checkpoint selection",
|
| 333 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 334 |
+
"row-level claim eligibility is false",
|
| 335 |
+
"shared recognizer still needs per-language slice reporting",
|
| 336 |
+
"packaged local model card is absent"
|
| 337 |
+
]
|
| 338 |
+
},
|
| 339 |
+
{
|
| 340 |
+
"id": "hf-recognizer-next-v1-model-card-smoke",
|
| 341 |
+
"task": "recognizer_oracle_crop",
|
| 342 |
+
"language": "nepali_mixed",
|
| 343 |
+
"script": "Devanagari",
|
| 344 |
+
"model_family": "hf-recognizer-next-v1",
|
| 345 |
+
"runner": {
|
| 346 |
+
"type": "model_config",
|
| 347 |
+
"model_config": "outputs/aws-g6-hf-recognizer-next-v1/package/model-card-cpu.json",
|
| 348 |
+
"training_summary": "outputs/aws-g6-hf-recognizer-next-v1/runs-hf-recognizer-next-v1/recognizer-training-summary.json"
|
| 349 |
+
},
|
| 350 |
+
"eval_pack": {
|
| 351 |
+
"label_file": "outputs/eval-packs/gorkhapatra-devanagari-real-v1/eval.txt",
|
| 352 |
+
"data_dir": "outputs/eval-packs/gorkhapatra-devanagari-real-v1",
|
| 353 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 354 |
+
"summary_file": "outputs/eval-packs/gorkhapatra-devanagari-real-v1/summary.json",
|
| 355 |
+
"dataset": "gorkhapatra-devanagari-real",
|
| 356 |
+
"domain": "real_print",
|
| 357 |
+
"gt_status": "machine_converted",
|
| 358 |
+
"split": "eval",
|
| 359 |
+
"eval_pack_version": "gorkhapatra-devanagari-real-v1",
|
| 360 |
+
"frozen_eval_pack": true,
|
| 361 |
+
"unseen_for_selection": false,
|
| 362 |
+
"claim_eligible": false,
|
| 363 |
+
"manifest_file": "outputs/eval-packs/gorkhapatra-devanagari-real-v1/manifests/recognizer-eval.jsonl"
|
| 364 |
+
},
|
| 365 |
+
"leakage_sources": [
|
| 366 |
+
{
|
| 367 |
+
"manifest_file": "data/splits-recognizer-next-v1/train.jsonl",
|
| 368 |
+
"split": "train"
|
| 369 |
+
},
|
| 370 |
+
{
|
| 371 |
+
"manifest_file": "data/splits-recognizer-next-v1/eval.jsonl",
|
| 372 |
+
"split": "dev"
|
| 373 |
+
}
|
| 374 |
+
],
|
| 375 |
+
"claim_blockers": [
|
| 376 |
+
"canonical eval pack audit is not claim-ready; see outputs/eval-packs/gorkhapatra-devanagari-real-v1/audit/recognizer-eval-pack-audit.json",
|
| 377 |
+
"model is generic mixed recognizer, not a per-language claim model",
|
| 378 |
+
"eval pack not declared unseen for model/checkpoint selection",
|
| 379 |
+
"GT is machine-converted and lacks native-reader calibration"
|
| 380 |
+
]
|
| 381 |
+
},
|
| 382 |
+
{
|
| 383 |
+
"id": "paddle-recognizer-next-v1-model-card-smoke",
|
| 384 |
+
"task": "recognizer_oracle_crop",
|
| 385 |
+
"language": "nepali_mixed",
|
| 386 |
+
"script": "Devanagari",
|
| 387 |
+
"model_family": "paddle-recognizer-next-v1",
|
| 388 |
+
"runner": {
|
| 389 |
+
"type": "model_config",
|
| 390 |
+
"model_config": "outputs/aws-g6-paddle-recognizer-next-v1/slim/runs/paddle-recognizer-next-v1/package/model-card.json",
|
| 391 |
+
"training_summary": "outputs/aws-g6-paddle-recognizer-next-v1/slim/runs/paddle-recognizer-next-v1/recognizer-training-summary.json"
|
| 392 |
+
},
|
| 393 |
+
"eval_pack": {
|
| 394 |
+
"label_file": "outputs/eval-packs/gorkhapatra-devanagari-real-v1/eval.txt",
|
| 395 |
+
"data_dir": "outputs/eval-packs/gorkhapatra-devanagari-real-v1",
|
| 396 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 397 |
+
"summary_file": "outputs/eval-packs/gorkhapatra-devanagari-real-v1/summary.json",
|
| 398 |
+
"dataset": "gorkhapatra-devanagari-real",
|
| 399 |
+
"domain": "real_print",
|
| 400 |
+
"gt_status": "machine_converted",
|
| 401 |
+
"split": "eval",
|
| 402 |
+
"eval_pack_version": "gorkhapatra-devanagari-real-v1",
|
| 403 |
+
"frozen_eval_pack": true,
|
| 404 |
+
"unseen_for_selection": false,
|
| 405 |
+
"claim_eligible": false,
|
| 406 |
+
"manifest_file": "outputs/eval-packs/gorkhapatra-devanagari-real-v1/manifests/recognizer-eval.jsonl"
|
| 407 |
+
},
|
| 408 |
+
"leakage_sources": [
|
| 409 |
+
{
|
| 410 |
+
"manifest_file": "data/splits-recognizer-next-v1/train.jsonl",
|
| 411 |
+
"split": "train"
|
| 412 |
+
},
|
| 413 |
+
{
|
| 414 |
+
"manifest_file": "data/splits-recognizer-next-v1/eval.jsonl",
|
| 415 |
+
"split": "dev"
|
| 416 |
+
}
|
| 417 |
+
],
|
| 418 |
+
"claim_blockers": [
|
| 419 |
+
"canonical eval pack audit is not claim-ready; see outputs/eval-packs/gorkhapatra-devanagari-real-v1/audit/recognizer-eval-pack-audit.json",
|
| 420 |
+
"model is generic mixed recognizer, not a per-language claim model",
|
| 421 |
+
"eval pack not declared unseen for model/checkpoint selection",
|
| 422 |
+
"GT is machine-converted and lacks native-reader calibration"
|
| 423 |
+
]
|
| 424 |
+
},
|
| 425 |
+
{
|
| 426 |
+
"id": "limbu-sirijonga-gorkhapatra-page-current-router",
|
| 427 |
+
"task": "page_level_ocr",
|
| 428 |
+
"language": "limbu",
|
| 429 |
+
"script": "Limbu",
|
| 430 |
+
"model_family": "limbu-first-router-v1",
|
| 431 |
+
"runner": {
|
| 432 |
+
"type": "external_command",
|
| 433 |
+
"command_template": "uv run python tools/copy_page_ocr_document.py --source-documents-dir outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/operational-image-line-width480-minwidth005-conf070-documents --sample-id {sample_id} --input-path {input_path} --out {output_dir} --document-json {document_json} --force",
|
| 434 |
+
"requires_operator_resolution": false,
|
| 435 |
+
"model_name": "operational-image-line-width480-minwidth005-conf070",
|
| 436 |
+
"checkpoint": "outputs/limbu-first-router-v1/components/limbu-sirijonga-paddle-v4/artifacts/inference.pdiparams",
|
| 437 |
+
"config": "outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/recognizer-width-ablation-v1/model-cards/input-width-480/model-card.json",
|
| 438 |
+
"dictionary": "outputs/limbu-first-router-v1/components/limbu-sirijonga-paddle-v4/artifacts/limbu-sirijonga-v2.txt",
|
| 439 |
+
"selected_checkpoint_on_test": false,
|
| 440 |
+
"notes": "Replays the current operational Sirijonga page OCR documents into the recipe runner. This makes the page gate reproducible from local artifacts, but it is not fresh OCR inference."
|
| 441 |
+
},
|
| 442 |
+
"eval_pack": {
|
| 443 |
+
"eval_manifest": "outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/manifests/eval-current-schema-gorkhapatra-sirijonga.jsonl",
|
| 444 |
+
"ocr_output_kind": "sidecar_fixture",
|
| 445 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 446 |
+
"summary_file": "outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/summary-current-schema-input.json",
|
| 447 |
+
"script_inventory_file": "outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/script-inventory.json",
|
| 448 |
+
"dataset": "gorkhapatra-limbu-sirijonga-page-draft",
|
| 449 |
+
"domain": "real_print",
|
| 450 |
+
"gt_status": "machine_converted_unreviewed",
|
| 451 |
+
"split": "eval",
|
| 452 |
+
"eval_pack_version": "gorkhapatra-limbu-sirijonga-page-draft-v1-current-schema-input",
|
| 453 |
+
"frozen_eval_pack": false,
|
| 454 |
+
"unseen_for_selection": true,
|
| 455 |
+
"claim_eligible": false
|
| 456 |
+
},
|
| 457 |
+
"leakage_sources": [
|
| 458 |
+
{
|
| 459 |
+
"manifest_file": "outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/train-dev-empty.jsonl",
|
| 460 |
+
"split": "train_dev_placeholder"
|
| 461 |
+
}
|
| 462 |
+
],
|
| 463 |
+
"claim_blockers": [
|
| 464 |
+
"page reference is machine-converted and unreviewed",
|
| 465 |
+
"page eval pack is not frozen",
|
| 466 |
+
"row-level claim eligibility is false",
|
| 467 |
+
"current full-page OCR has unresolved review rows and weak detector/layout recall",
|
| 468 |
+
"recipe replays pre-materialized operational OCR documents rather than running fresh page OCR"
|
| 469 |
+
]
|
| 470 |
+
},
|
| 471 |
+
{
|
| 472 |
+
"id": "limbu-sirijonga-gorkhapatra-book-conf070-coverage-gated",
|
| 473 |
+
"task": "page_level_ocr",
|
| 474 |
+
"language": "limbu",
|
| 475 |
+
"script": "Limbu",
|
| 476 |
+
"model_family": "limbu-sirijonga-gorkhapatra-conf070-book-replay",
|
| 477 |
+
"runner": {
|
| 478 |
+
"type": "external_command",
|
| 479 |
+
"command_template": "uv run python tools/score_limbu_book_ocr_candidate.py --book-output outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/gorkhapatra-operational-conf070-book-replay --book-audit outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/gorkhapatra-operational-conf070-book-replay/limbu-book-output-audit.json --documents-dir outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/gorkhapatra-operational-conf070-book-replay/score-documents --eval-manifest outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/gorkhapatra-operational-conf070-book-sidecar-replay-inputs/eval-replay.jsonl --ocr-output-kind sidecar_fixture --summary-file outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/summary-current-schema-input.json --script-inventory-file outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/script-inventory.json --train-manifest outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/train-dev-empty.jsonl --system-name limbu-sirijonga-gorkhapatra-conf070-book-replay --model-name operational-image-line-width480-minwidth005-conf070 --checkpoint outputs/limbu-first-router-v1/components/limbu-sirijonga-paddle-v4/artifacts/inference.pdiparams --config outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/recognizer-width-ablation-v1/model-cards/input-width-480/model-card.json --dictionary outputs/limbu-first-router-v1/components/limbu-sirijonga-paddle-v4/artifacts/limbu-sirijonga-v2.txt --language limbu --script Limbu --domain real_print --gt-status machine_converted_unreviewed --eval-pack-version gorkhapatra-limbu-sirijonga-page-draft-v1-current-schema-input --no-frozen-eval-pack --unseen-for-selection --no-selected-checkpoint-on-test --no-claim-eligible --line-iou-threshold 0.5 --review-sample-size 50 --champion-metrics outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/score-gorkhapatra-operational-conf070-review-application-pending/metrics.json --review-coverage-report outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/gorkhapatra-operational-conf070-book-missed-line-review-coverage-with-missed-refs/missed-line-review-coverage.json --baseline-codepoint-cer 0.7393608074011775 --baseline-grapheme-cer 0.7682623819778035 --baseline-wer 0.9189686924493554 --min-codepoint-improvement 0 --max-grapheme-regression 0 --max-wer-regression 0 --max-champion-codepoint-regression 0 --max-champion-grapheme-regression 0 --max-champion-wer-regression 0 --max-review-uncovered-or-weak-count 0 --max-review-extra-uncovered-or-weak-count 0 --min-line-f1 0.87 --min-reading-order-accuracy 0.77 --max-blank-line-rate 0 --max-extra-line-rate 0.02 --max-missed-line-rate 0.23 --min-scored-pages 1 --min-claim-pages 30 --out {output_dir} # input {input_path}",
|
| 480 |
+
"requires_operator_resolution": false,
|
| 481 |
+
"notes": "Sirijonga book-page replay of the current operational conf070 OCR document. The command scores the materialized book projection and gates it against the selected Sirijonga diagnostic plus unresolved-line coverage."
|
| 482 |
+
},
|
| 483 |
+
"eval_pack": {
|
| 484 |
+
"eval_manifest": "outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/gorkhapatra-operational-conf070-book-sidecar-replay-inputs/eval-replay.jsonl",
|
| 485 |
+
"ocr_output_kind": "sidecar_fixture",
|
| 486 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 487 |
+
"summary_file": "outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/summary-current-schema-input.json",
|
| 488 |
+
"script_inventory_file": "outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/script-inventory.json",
|
| 489 |
+
"dataset": "gorkhapatra-limbu-sirijonga-page-draft",
|
| 490 |
+
"domain": "real_print",
|
| 491 |
+
"gt_status": "machine_converted_unreviewed",
|
| 492 |
+
"split": "eval",
|
| 493 |
+
"eval_pack_version": "gorkhapatra-limbu-sirijonga-page-draft-v1-current-schema-input",
|
| 494 |
+
"frozen_eval_pack": false,
|
| 495 |
+
"unseen_for_selection": true,
|
| 496 |
+
"claim_eligible": false
|
| 497 |
+
},
|
| 498 |
+
"evidence": {
|
| 499 |
+
"book_output": "outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/gorkhapatra-operational-conf070-book-replay",
|
| 500 |
+
"score_admission": "outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/score-admit-gorkhapatra-operational-conf070-book-replay-coverage-gated/limbu-book-ocr-score-admission.json",
|
| 501 |
+
"admission_report": "outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/score-admit-gorkhapatra-operational-conf070-book-replay-coverage-gated/admission/limbu-book-ocr-candidate-admission.json",
|
| 502 |
+
"metrics": "outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/score-admit-gorkhapatra-operational-conf070-book-replay-coverage-gated/score/metrics.json",
|
| 503 |
+
"review_coverage_report": "outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/gorkhapatra-operational-conf070-book-missed-line-review-coverage-with-missed-refs/missed-line-review-coverage.json",
|
| 504 |
+
"unresolved_review_packet": "outputs/limbu-sirijonga-page-unresolved-review-2026-06-22/gorkhapatra-operational-conf070-book-unresolved-review-packet-with-missed-refs/book-unresolved-review-packet.tsv"
|
| 505 |
+
},
|
| 506 |
+
"leakage_sources": [
|
| 507 |
+
{
|
| 508 |
+
"manifest_file": "outputs/limbu-page-eval-drafts/gorkhapatra-limbu-sirijonga-2026-06-21/train-dev-empty.jsonl",
|
| 509 |
+
"split": "train_dev_placeholder"
|
| 510 |
+
}
|
| 511 |
+
],
|
| 512 |
+
"claim_blockers": [
|
| 513 |
+
"page reference is machine-converted and unreviewed",
|
| 514 |
+
"page eval pack is not frozen",
|
| 515 |
+
"row-level claim eligibility is false",
|
| 516 |
+
"claim_safe=false because the score has claim_eligible_rows=0 and only 1 scored page",
|
| 517 |
+
"79 unresolved review packet rows still require reviewed correction or safe automatic recovery",
|
| 518 |
+
"Sirijonga candidate is operationally admitted only for the current 1-page Gorkhapatra draft set"
|
| 519 |
+
]
|
| 520 |
+
},
|
| 521 |
+
{
|
| 522 |
+
"id": "limbu-devanagari-cdc-page-live-tesseract",
|
| 523 |
+
"task": "page_level_ocr",
|
| 524 |
+
"language": "limbu",
|
| 525 |
+
"script": "Devanagari",
|
| 526 |
+
"model_family": "limbu-devanagari-cdc-live-page-baseline",
|
| 527 |
+
"runner": {
|
| 528 |
+
"type": "external_command",
|
| 529 |
+
"command_template": "REPLACE_WITH_LIMBU_DEVANAGARI_PAGE_OCR_COMMAND --input {input_path} --out {output_dir}",
|
| 530 |
+
"requires_operator_resolution": true,
|
| 531 |
+
"notes": "Tracks the CDC Devanagari-written Limbu full-page draft gate; Tesseract is a failed baseline until the trained recognizer is admitted.",
|
| 532 |
+
"output_audit": {
|
| 533 |
+
"command_template": "uv run ocrtech audit-limbu-output {output_dir} --require-capture-prep --min-capture-prepared-width 1000 --min-capture-prepared-height 1000 --min-capture-prepared-entropy 0.5 --min-capture-prepared-luminance-stddev 1.0 --min-capture-prepared-edge-stddev 1.0 --min-line-count 30 --min-average-line-confidence 0.80 --min-quality-score 0.85 --require-no-pending-review --require-no-dropped-image-lines --require-script devanagari_limbu",
|
| 534 |
+
"required_flags": [
|
| 535 |
+
"--require-capture-prep",
|
| 536 |
+
"--min-capture-prepared-width",
|
| 537 |
+
"--min-capture-prepared-height",
|
| 538 |
+
"--min-capture-prepared-entropy",
|
| 539 |
+
"--min-capture-prepared-luminance-stddev",
|
| 540 |
+
"--min-capture-prepared-edge-stddev",
|
| 541 |
+
"--min-line-count",
|
| 542 |
+
"--min-average-line-confidence",
|
| 543 |
+
"--min-quality-score",
|
| 544 |
+
"--require-no-pending-review",
|
| 545 |
+
"--require-no-dropped-image-lines"
|
| 546 |
+
],
|
| 547 |
+
"required_scripts": [
|
| 548 |
+
"devanagari_limbu"
|
| 549 |
+
]
|
| 550 |
+
}
|
| 551 |
+
},
|
| 552 |
+
"eval_pack": {
|
| 553 |
+
"eval_manifest": "outputs/limbu-page-eval-drafts/limbu-devanagari-cdc-curriculum-linebox-2026-06-21/manifests/eval.jsonl",
|
| 554 |
+
"ocr_output_kind": "live_detector",
|
| 555 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 556 |
+
"summary_file": "outputs/limbu-page-eval-drafts/limbu-devanagari-cdc-curriculum-linebox-2026-06-21/summary.json",
|
| 557 |
+
"script_inventory_file": "outputs/limbu-page-eval-drafts/limbu-devanagari-cdc-curriculum-linebox-2026-06-21/script-inventory.json",
|
| 558 |
+
"dataset": "limbu-devanagari-cdc-curriculum-page-draft",
|
| 559 |
+
"domain": "real_print",
|
| 560 |
+
"gt_status": "curriculum_text_unreviewed",
|
| 561 |
+
"split": "eval",
|
| 562 |
+
"eval_pack_version": "limbu-devanagari-cdc-curriculum-linebox-draft-v1",
|
| 563 |
+
"frozen_eval_pack": false,
|
| 564 |
+
"unseen_for_selection": true,
|
| 565 |
+
"claim_eligible": false
|
| 566 |
+
},
|
| 567 |
+
"leakage_sources": [
|
| 568 |
+
{
|
| 569 |
+
"manifest_file": "outputs/limbu-page-eval-drafts/limbu-devanagari-cdc-curriculum-linebox-2026-06-21/train-dev-empty.jsonl",
|
| 570 |
+
"split": "train_dev_placeholder"
|
| 571 |
+
}
|
| 572 |
+
],
|
| 573 |
+
"claim_blockers": [
|
| 574 |
+
"CDC page references are unreviewed curriculum/PDF-aligned draft evidence",
|
| 575 |
+
"page eval pack is not frozen",
|
| 576 |
+
"row-level claim eligibility is false",
|
| 577 |
+
"Tesseract baseline is not an adequate Devanagari-written Limbu primary",
|
| 578 |
+
"reproducible page OCR command must be resolved after recognizer admission"
|
| 579 |
+
]
|
| 580 |
+
},
|
| 581 |
+
{
|
| 582 |
+
"id": "limbu-devanagari-cdc-page-marker-v2-champion-coverage-gated",
|
| 583 |
+
"task": "page_level_ocr",
|
| 584 |
+
"language": "limbu",
|
| 585 |
+
"script": "Devanagari",
|
| 586 |
+
"model_family": "limbu-devanagari-cdc-ppocrv5-page-marker-v2",
|
| 587 |
+
"runner": {
|
| 588 |
+
"type": "external_command",
|
| 589 |
+
"command_template": "uv run python tools/score_limbu_book_ocr_candidate.py --book-output outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/book-line-split-recognizer-prefill-line-text-complete-groups-structural-order-page-marker-v2-application --book-audit outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/book-line-split-recognizer-prefill-line-text-complete-groups-structural-order-page-marker-v2-application/candidate-line-application-audit.json --champion-metrics outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/score-admit-book-line-split-recognizer-prefill-line-text-complete-groups-structural-order-page-marker-v2/score/metrics.json --review-coverage-report outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/missed-line-review-coverage-page-marker-v2-current-best-packet/missed-line-review-coverage.json --out {output_dir} --documents-dir {documents_dir} # input {input_path}",
|
| 590 |
+
"requires_operator_resolution": false,
|
| 591 |
+
"notes": "Current Devanagari-written Limbu book-page candidate. The command scores the materialized book projection and applies old-baseline, champion non-regression, and unresolved-line coverage gates.",
|
| 592 |
+
"output_audit": {
|
| 593 |
+
"command_template": "uv run python tools/audit_candidate_line_review_application.py --application-dir {output_dir}",
|
| 594 |
+
"required_flags": [
|
| 595 |
+
"--require-capture-prep",
|
| 596 |
+
"--min-capture-prepared-width",
|
| 597 |
+
"--min-capture-prepared-height",
|
| 598 |
+
"--min-capture-prepared-entropy",
|
| 599 |
+
"--min-capture-prepared-luminance-stddev",
|
| 600 |
+
"--min-capture-prepared-edge-stddev",
|
| 601 |
+
"--min-line-count",
|
| 602 |
+
"--min-average-line-confidence",
|
| 603 |
+
"--min-quality-score",
|
| 604 |
+
"--require-no-pending-review",
|
| 605 |
+
"--require-no-dropped-image-lines"
|
| 606 |
+
],
|
| 607 |
+
"required_scripts": [
|
| 608 |
+
"devanagari_limbu"
|
| 609 |
+
]
|
| 610 |
+
}
|
| 611 |
+
},
|
| 612 |
+
"eval_pack": {
|
| 613 |
+
"eval_manifest": "outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/multi-page-cdc-h20-structural-replay-inputs/eval-replay.jsonl",
|
| 614 |
+
"ocr_output_kind": "sidecar_fixture",
|
| 615 |
+
"normalization_profile": "NFC_preserve_whitespace",
|
| 616 |
+
"summary_file": "outputs/limbu-page-eval-drafts/limbu-devanagari-cdc-curriculum-linebox-2026-06-21/summary.json",
|
| 617 |
+
"script_inventory_file": "outputs/limbu-page-eval-drafts/limbu-devanagari-cdc-curriculum-linebox-2026-06-21/script-inventory.json",
|
| 618 |
+
"dataset": "limbu-devanagari-cdc-curriculum-page-draft",
|
| 619 |
+
"domain": "real_print",
|
| 620 |
+
"gt_status": "curriculum_text_unreviewed",
|
| 621 |
+
"split": "eval",
|
| 622 |
+
"eval_pack_version": "limbu-devanagari-cdc-curriculum-linebox-draft-v1",
|
| 623 |
+
"frozen_eval_pack": false,
|
| 624 |
+
"unseen_for_selection": true,
|
| 625 |
+
"claim_eligible": false
|
| 626 |
+
},
|
| 627 |
+
"evidence": {
|
| 628 |
+
"book_output": "outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/book-line-split-recognizer-prefill-line-text-complete-groups-structural-order-page-marker-v2-application",
|
| 629 |
+
"score_admission": "outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/score-admit-book-line-split-recognizer-prefill-line-text-complete-groups-structural-order-page-marker-v2-champion-coverage-gated/limbu-book-ocr-score-admission.json",
|
| 630 |
+
"admission_report": "outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/score-admit-book-line-split-recognizer-prefill-line-text-complete-groups-structural-order-page-marker-v2-champion-coverage-gated/admission/limbu-book-ocr-candidate-admission.json",
|
| 631 |
+
"metrics": "outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/score-admit-book-line-split-recognizer-prefill-line-text-complete-groups-structural-order-page-marker-v2-champion-coverage-gated/score/metrics.json",
|
| 632 |
+
"review_coverage_report": "outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/missed-line-review-coverage-page-marker-v2-current-best-packet/missed-line-review-coverage.json",
|
| 633 |
+
"unresolved_review_packet": "outputs/limbu-book-ocr-missed-line-recovery-2026-06-22/book-unresolved-review-packet-page-marker-v2-current-best-extra-predictions/book-unresolved-review-packet.tsv"
|
| 634 |
+
},
|
| 635 |
+
"leakage_sources": [
|
| 636 |
+
{
|
| 637 |
+
"manifest_file": "outputs/limbu-page-eval-drafts/limbu-devanagari-cdc-curriculum-linebox-2026-06-21/train-dev-empty.jsonl",
|
| 638 |
+
"split": "train_dev_placeholder"
|
| 639 |
+
}
|
| 640 |
+
],
|
| 641 |
+
"claim_blockers": [
|
| 642 |
+
"CDC page references are unreviewed curriculum/PDF-aligned draft evidence",
|
| 643 |
+
"page eval pack is not frozen",
|
| 644 |
+
"row-level claim eligibility is false",
|
| 645 |
+
"claim_safe=false because the score has claim_eligible_rows=0 and only 3 scored pages",
|
| 646 |
+
"18 unresolved review rows still require reviewed correction or safe automatic recovery",
|
| 647 |
+
"Devanagari candidate is operationally admitted only for the current 3-page CDC draft set"
|
| 648 |
+
]
|
| 649 |
+
}
|
| 650 |
+
]
|
| 651 |
+
}
|
ocr-tech-mps-integration/configs/eval-target-coverage-v1.json
ADDED
|
@@ -0,0 +1,193 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"version": "eval-target-coverage-v1",
|
| 3 |
+
"updated": "2026-06-22",
|
| 4 |
+
"scope_note": "Tracks intended Nepal/diaspora OCR evaluation coverage. Claim readiness is determined by linked eval-pack normalization recipes, eval run recipes, and their blockers; target presence alone is not evidence.",
|
| 5 |
+
"target_sets": [
|
| 6 |
+
{
|
| 7 |
+
"id": "limbu-first-two-script-ocr",
|
| 8 |
+
"description": "Required coverage for the Limbu-first OCR pipeline: Sirijonga-script Limbu and Devanagari-written Limbu must both be represented by claim-ready eval harness entries before any two-script Limbu OCR claim.",
|
| 9 |
+
"required_for_claim": true,
|
| 10 |
+
"requires_page_eval_recipe": true,
|
| 11 |
+
"requires_book_candidate_evidence": true,
|
| 12 |
+
"targets": [
|
| 13 |
+
{
|
| 14 |
+
"language": "limbu",
|
| 15 |
+
"display_name": "Limbu/Yakthung",
|
| 16 |
+
"script": "Limbu",
|
| 17 |
+
"first_script_target": "Limbu/Sirijonga",
|
| 18 |
+
"eval_status": "registered_real_pack",
|
| 19 |
+
"normalization_pack_id": "limbu-gorkhapatra-real-v1",
|
| 20 |
+
"eval_recipe_id": "limbu-sirijonga-gorkhapatra-oracle-crop",
|
| 21 |
+
"page_eval_recipe_id": "limbu-sirijonga-gorkhapatra-book-conf070-coverage-gated",
|
| 22 |
+
"notes": "Tracks the current real Sirijonga recognizer pack and the operationally admitted one-page book replay. Still blocked by machine-converted GT, row claim eligibility, one-page sample size, unseen-selection status, and missing packaged model-card evidence."
|
| 23 |
+
},
|
| 24 |
+
{
|
| 25 |
+
"language": "limbu",
|
| 26 |
+
"display_name": "Limbu/Yakthung",
|
| 27 |
+
"script": "Devanagari",
|
| 28 |
+
"first_script_target": "Devanagari-written Limbu",
|
| 29 |
+
"eval_status": "registered_real_pack",
|
| 30 |
+
"normalization_pack_id": "limbu-devanagari-cdc-line-crop-corpus-draft-v1",
|
| 31 |
+
"eval_recipe_id": "limbu-devanagari-cdc-ppocrv5-devclean-heldout",
|
| 32 |
+
"page_eval_recipe_id": "limbu-devanagari-cdc-page-marker-v2-champion-coverage-gated",
|
| 33 |
+
"notes": "Tracks the current Devanagari-written Limbu CDC book-page candidate: page-marker-v2 with champion non-regression and unresolved-line coverage admission. It is still not claim-ready until the pack is frozen and text/line references are reviewed."
|
| 34 |
+
}
|
| 35 |
+
]
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"id": "first-ten-document-ocr",
|
| 39 |
+
"description": "First ten language targets from docs/NEPAL_MULTILINGUAL_OCR_GOAL.md.",
|
| 40 |
+
"required_for_claim": false,
|
| 41 |
+
"targets": [
|
| 42 |
+
{
|
| 43 |
+
"language": "limbu",
|
| 44 |
+
"display_name": "Limbu/Yakthung",
|
| 45 |
+
"script": "Limbu",
|
| 46 |
+
"first_script_target": "Limbu/Sirijonga",
|
| 47 |
+
"eval_status": "partial_real_recognizer_pack",
|
| 48 |
+
"normalization_pack_id": "limbu-gorkhapatra-real-v1",
|
| 49 |
+
"eval_recipe_id": "limbu-sirijonga-gorkhapatra-oracle-crop",
|
| 50 |
+
"notes": "First non-Devanagari target; current pack lacks row provenance and native-reader calibration."
|
| 51 |
+
},
|
| 52 |
+
{
|
| 53 |
+
"language": "nepali",
|
| 54 |
+
"display_name": "Nepali",
|
| 55 |
+
"script": "Devanagari",
|
| 56 |
+
"first_script_target": "Devanagari",
|
| 57 |
+
"eval_status": "shared_devanagari_pack",
|
| 58 |
+
"normalization_pack_id": "gorkhapatra-devanagari-real-v1",
|
| 59 |
+
"eval_recipe_id": "devanagari-shared-gorkhapatra-oracle-crop",
|
| 60 |
+
"notes": "Core baseline language; current shared pack still needs per-language row slices for a Nepali-only claim."
|
| 61 |
+
},
|
| 62 |
+
{
|
| 63 |
+
"language": "english",
|
| 64 |
+
"display_name": "English",
|
| 65 |
+
"script": "Latin",
|
| 66 |
+
"first_script_target": "Latin",
|
| 67 |
+
"eval_status": "missing_real_eval_pack",
|
| 68 |
+
"notes": "Must not regress for mixed documents, but no claim-grade recognizer eval pack is registered here."
|
| 69 |
+
},
|
| 70 |
+
{
|
| 71 |
+
"language": "tamang",
|
| 72 |
+
"display_name": "Tamang",
|
| 73 |
+
"script": "Devanagari",
|
| 74 |
+
"first_script_target": "Devanagari first; Tamyig audit later",
|
| 75 |
+
"eval_status": "missing_real_eval_pack",
|
| 76 |
+
"notes": "Local corpus exists per project docs; no registered real OCR eval pack yet."
|
| 77 |
+
},
|
| 78 |
+
{
|
| 79 |
+
"language": "magar",
|
| 80 |
+
"display_name": "Magar",
|
| 81 |
+
"script": "Devanagari",
|
| 82 |
+
"first_script_target": "Devanagari first",
|
| 83 |
+
"eval_status": "missing_real_eval_pack",
|
| 84 |
+
"notes": "Gorkhapatra item confirmed in project docs; no registered real OCR eval pack yet."
|
| 85 |
+
},
|
| 86 |
+
{
|
| 87 |
+
"language": "wambule",
|
| 88 |
+
"display_name": "Wambule",
|
| 89 |
+
"script": "Devanagari",
|
| 90 |
+
"first_script_target": "Devanagari first",
|
| 91 |
+
"eval_status": "missing_real_eval_pack",
|
| 92 |
+
"notes": "Gorkhapatra item confirmed in project docs; no registered real OCR eval pack yet."
|
| 93 |
+
},
|
| 94 |
+
{
|
| 95 |
+
"language": "bantawa_rai",
|
| 96 |
+
"display_name": "Bantawa/Rai",
|
| 97 |
+
"script": "Devanagari",
|
| 98 |
+
"first_script_target": "Devanagari or Kirat Rai depending source",
|
| 99 |
+
"eval_status": "missing_real_eval_pack",
|
| 100 |
+
"notes": "Rai-family target; Kirat Rai native-script pack is tracked separately as current real-pack coverage."
|
| 101 |
+
},
|
| 102 |
+
{
|
| 103 |
+
"language": "kulung",
|
| 104 |
+
"display_name": "Kulung",
|
| 105 |
+
"script": "Devanagari",
|
| 106 |
+
"first_script_target": "Devanagari first",
|
| 107 |
+
"eval_status": "missing_real_eval_pack",
|
| 108 |
+
"notes": "Gorkhapatra item confirmed in project docs; no registered real OCR eval pack yet."
|
| 109 |
+
},
|
| 110 |
+
{
|
| 111 |
+
"language": "yakkha",
|
| 112 |
+
"display_name": "Yakkha",
|
| 113 |
+
"script": "Devanagari",
|
| 114 |
+
"first_script_target": "Devanagari first",
|
| 115 |
+
"eval_status": "missing_real_eval_pack",
|
| 116 |
+
"notes": "Gorkhapatra item confirmed in project docs; no registered real OCR eval pack yet."
|
| 117 |
+
},
|
| 118 |
+
{
|
| 119 |
+
"language": "tharu_rana",
|
| 120 |
+
"display_name": "Tharu/Rana Tharu",
|
| 121 |
+
"script": "Devanagari",
|
| 122 |
+
"first_script_target": "Devanagari first",
|
| 123 |
+
"eval_status": "missing_real_eval_pack",
|
| 124 |
+
"notes": "Multiple varieties; labels must remain explicit before any claim."
|
| 125 |
+
}
|
| 126 |
+
]
|
| 127 |
+
},
|
| 128 |
+
{
|
| 129 |
+
"id": "current-real-recognizer-packs",
|
| 130 |
+
"description": "Current real recognizer eval packs registered for the harness.",
|
| 131 |
+
"required_for_claim": true,
|
| 132 |
+
"targets": [
|
| 133 |
+
{
|
| 134 |
+
"language": "newa",
|
| 135 |
+
"display_name": "Newa / Nepal Bhasa",
|
| 136 |
+
"script": "Newa",
|
| 137 |
+
"first_script_target": "Prachalit/Newa",
|
| 138 |
+
"eval_status": "registered_real_pack",
|
| 139 |
+
"normalization_pack_id": "newa-zenodo-6967421-v1",
|
| 140 |
+
"eval_recipe_id": "newa-zenodo-native-retrain-oracle-crop"
|
| 141 |
+
},
|
| 142 |
+
{
|
| 143 |
+
"language": "sunuwar",
|
| 144 |
+
"display_name": "Sunuwar / Jenticha",
|
| 145 |
+
"script": "Sunuwar",
|
| 146 |
+
"first_script_target": "Jenticha/Sunuwar",
|
| 147 |
+
"eval_status": "registered_real_pack",
|
| 148 |
+
"normalization_pack_id": "sunuwar-herald-real-v1",
|
| 149 |
+
"eval_recipe_id": "sunuwar-jenticha-samefont-oracle-crop"
|
| 150 |
+
},
|
| 151 |
+
{
|
| 152 |
+
"language": "lepcha",
|
| 153 |
+
"display_name": "Lepcha",
|
| 154 |
+
"script": "Lepcha",
|
| 155 |
+
"first_script_target": "Lepcha",
|
| 156 |
+
"eval_status": "registered_real_pack",
|
| 157 |
+
"normalization_pack_id": "lepcha-herald-real-v1",
|
| 158 |
+
"eval_recipe_id": "lepcha-herald-real-oracle-crop",
|
| 159 |
+
"notes": "Eval recipe exists, but model command, GT trust, and row claim eligibility blockers remain."
|
| 160 |
+
},
|
| 161 |
+
{
|
| 162 |
+
"language": "limbu",
|
| 163 |
+
"display_name": "Limbu/Yakthung",
|
| 164 |
+
"script": "Limbu",
|
| 165 |
+
"first_script_target": "Limbu/Sirijonga",
|
| 166 |
+
"eval_status": "registered_real_pack",
|
| 167 |
+
"normalization_pack_id": "limbu-gorkhapatra-real-v1",
|
| 168 |
+
"eval_recipe_id": "limbu-sirijonga-gorkhapatra-oracle-crop"
|
| 169 |
+
},
|
| 170 |
+
{
|
| 171 |
+
"language": "kiratrai",
|
| 172 |
+
"display_name": "Kirat Rai",
|
| 173 |
+
"script": "KiratRai",
|
| 174 |
+
"first_script_target": "Kirat Rai",
|
| 175 |
+
"eval_status": "registered_real_pack",
|
| 176 |
+
"normalization_pack_id": "kiratrai-herald-real-v1",
|
| 177 |
+
"eval_recipe_id": "kiratrai-herald-real-oracle-crop",
|
| 178 |
+
"notes": "Eval recipe exists, but model command, GT trust, provenance, and row claim eligibility blockers remain."
|
| 179 |
+
},
|
| 180 |
+
{
|
| 181 |
+
"language": "devanagari_shared",
|
| 182 |
+
"display_name": "Shared Devanagari Gorkhapatra",
|
| 183 |
+
"script": "Devanagari",
|
| 184 |
+
"first_script_target": "Shared Devanagari",
|
| 185 |
+
"eval_status": "registered_real_pack",
|
| 186 |
+
"normalization_pack_id": "gorkhapatra-devanagari-real-v1",
|
| 187 |
+
"eval_recipe_id": "devanagari-shared-gorkhapatra-oracle-crop",
|
| 188 |
+
"notes": "Shared pack must still be sliced into explicit language rows before supporting individual language claims."
|
| 189 |
+
}
|
| 190 |
+
]
|
| 191 |
+
}
|
| 192 |
+
]
|
| 193 |
+
}
|
ocr-tech-mps-integration/configs/limbu-devanagari-ppocrv5-admission-plan-v1.json
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"version": "limbu-devanagari-ppocrv5-admission-plan-v1",
|
| 3 |
+
"updated": "2026-06-21",
|
| 4 |
+
"plans": [
|
| 5 |
+
{
|
| 6 |
+
"id": "limbu-devanagari-cdc-ppocrv5-devclean-admission",
|
| 7 |
+
"language": "limbu",
|
| 8 |
+
"script": "Devanagari",
|
| 9 |
+
"model_family": "limbu-devanagari-cdc-ppocrv5-mobile",
|
| 10 |
+
"purpose": "Train a same-register Devanagari-written Limbu recognizer from real CDC line crops, select checkpoints only on the dev split, and reserve the 31-row eval split for a single held-out admission score after checkpoint selection.",
|
| 11 |
+
"training_bundle": {
|
| 12 |
+
"archive": "runs/limbu-devanagari-cdc-ppocrv5-mobile-training-bundle-2026-06-21/limbu-devanagari-cdc-ppocrv5-mobile-devclean-training-bundle.tar.gz",
|
| 13 |
+
"archive_sha256": "01b3689603309c13cb31ef31ff7dc0def65b556a92269d33483ee11da6e8c14d",
|
| 14 |
+
"manifest": "runs/limbu-devanagari-cdc-ppocrv5-mobile-training-bundle-2026-06-21/recognizer-training-bundle.json",
|
| 15 |
+
"manifest_sha256": "95322aa852b14662e22c53ddbc1ae8346f37a51a7d1ae8348ec4f7a5fbf8da9f",
|
| 16 |
+
"recipe_backend": "paddleocr",
|
| 17 |
+
"run_command_after_extract": "python PaddleOCR/tools/train.py -c training-recipes/ppocrv5-mobile-devclean/paddle_recognizer.yml",
|
| 18 |
+
"bundle_train_manifest": "manifests/train.jsonl",
|
| 19 |
+
"bundle_dev_manifest": "manifests/dev.jsonl"
|
| 20 |
+
},
|
| 21 |
+
"selection_policy": {
|
| 22 |
+
"train_manifest": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/manifests/train.jsonl",
|
| 23 |
+
"dev_manifest": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/manifests/dev.jsonl",
|
| 24 |
+
"train_plus_dev_manifest": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/train-manifest.jsonl",
|
| 25 |
+
"sealed_eval_manifest": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/manifests/recognizer-eval.jsonl",
|
| 26 |
+
"sealed_eval_label_file": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/eval.txt",
|
| 27 |
+
"allowed_checkpoint_selection_splits": [
|
| 28 |
+
"train",
|
| 29 |
+
"dev"
|
| 30 |
+
],
|
| 31 |
+
"forbidden_checkpoint_selection_splits": [
|
| 32 |
+
"eval"
|
| 33 |
+
],
|
| 34 |
+
"min_train_samples": 5000,
|
| 35 |
+
"min_dev_samples": 200,
|
| 36 |
+
"min_sealed_eval_samples": 31,
|
| 37 |
+
"expected_split_counts": {
|
| 38 |
+
"train": 5551,
|
| 39 |
+
"dev": 258,
|
| 40 |
+
"eval": 31,
|
| 41 |
+
"train_plus_dev": 5809
|
| 42 |
+
},
|
| 43 |
+
"required_no_overlap_fields": [
|
| 44 |
+
"sample_id",
|
| 45 |
+
"image_path",
|
| 46 |
+
"image_sha256",
|
| 47 |
+
"text_sha256",
|
| 48 |
+
"sample_sha256"
|
| 49 |
+
]
|
| 50 |
+
},
|
| 51 |
+
"admission_recipe": {
|
| 52 |
+
"registry": "configs/eval-harness-recipes-v1.json",
|
| 53 |
+
"recipe_id": "limbu-devanagari-cdc-ppocrv5-devclean-heldout",
|
| 54 |
+
"requires_unseen_for_selection": true,
|
| 55 |
+
"requires_selected_checkpoint_on_test": false,
|
| 56 |
+
"requires_eval_split": "eval",
|
| 57 |
+
"requires_leakage_source_manifest": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/train-manifest.jsonl",
|
| 58 |
+
"claim_blockers_until_resolved": [
|
| 59 |
+
"trained checkpoint path and reproducible prediction command are unresolved until the PP-OCRv5 bundle is run",
|
| 60 |
+
"eval pack is not frozen",
|
| 61 |
+
"GT is unreviewed curriculum text and lacks human/native-reader calibration",
|
| 62 |
+
"row-level claim eligibility is false",
|
| 63 |
+
"packaged local model card is absent"
|
| 64 |
+
]
|
| 65 |
+
}
|
| 66 |
+
}
|
| 67 |
+
]
|
| 68 |
+
}
|
ocr-tech-mps-integration/configs/limbu-first-ocr-pipeline-v1.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
ocr-tech-mps-integration/configs/limbu-sirijonga-augmentation-ablation-v1.json
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ablation_id": "limbu-sirijonga-augmentation-ablation-v1",
|
| 3 |
+
"language": "limbu",
|
| 4 |
+
"script": "Limbu / Sirijonga",
|
| 5 |
+
"purpose": "Compare clean-only, lighting-augmented, and camera-angle/phone-photo synthetic OCR line training for Limbu/Sirijonga as training ablations only.",
|
| 6 |
+
"claim_policy": {
|
| 7 |
+
"claim_evidence_eligible": false,
|
| 8 |
+
"reason": "All rows are rendered synthetic training or ablation data. Accuracy claims require held-out real Gorkhapatra/Naya Nepal references."
|
| 9 |
+
},
|
| 10 |
+
"source_bundle": {
|
| 11 |
+
"rendered_split": "outputs/sirijonga-limbu-rendered-split-namdhinggo-augmented-2026-06-16",
|
| 12 |
+
"font_readiness_report": "outputs/font-asset-readiness-namdhinggo-2026-06-16/font-asset-readiness.json",
|
| 13 |
+
"font_sha256": "1b46b16277e4b6784bd19306a9474281888b656e9c57233b4e7dc63bd1229d6c",
|
| 14 |
+
"degradation_seed": 29,
|
| 15 |
+
"full_split_audit": "outputs/sirijonga-limbu-rendered-split-namdhinggo-augmented-audit-2026-06-16/rendered-degradation-split-audit.json"
|
| 16 |
+
},
|
| 17 |
+
"dictionary": "data/dictionaries/limbu-sirijonga-v1.txt",
|
| 18 |
+
"backend": {
|
| 19 |
+
"name": "paddleocr",
|
| 20 |
+
"base_model": "https://paddle-model-ecology.bj.bcebos.com/paddlex/official_pretrained_model/devanagari_PP-OCRv3_mobile_rec_pretrained.pdparams",
|
| 21 |
+
"status": "narrow_limbu_ablation_only",
|
| 22 |
+
"guardrail": "The broader Devanagari PP-OCRv3 mixed recognizer family is stopped for engine promotion by prior admission evidence. These runs are allowed only as a Limbu/Sirijonga synthetic augmentation ablation and must not be promoted without real-document validation."
|
| 23 |
+
},
|
| 24 |
+
"training_defaults": {
|
| 25 |
+
"train_batch_size": 8,
|
| 26 |
+
"eval_batch_size": 8,
|
| 27 |
+
"learning_rate": 0.0001,
|
| 28 |
+
"epochs": 1
|
| 29 |
+
},
|
| 30 |
+
"arms": [
|
| 31 |
+
{
|
| 32 |
+
"name": "clean",
|
| 33 |
+
"profiles": ["clean"],
|
| 34 |
+
"train_manifest": "data/ablation-limbu-namdhinggo-augmented-v1/train-clean.jsonl",
|
| 35 |
+
"eval_manifest": "data/ablation-limbu-namdhinggo-augmented-v1/eval-clean.jsonl",
|
| 36 |
+
"audit": "outputs/ablation-limbu-namdhinggo-augmented-v1/audit-clean/rendered-degradation-split-audit.json",
|
| 37 |
+
"recipe_dir": "runs/sirijonga-limbu-paddle-ablation-clean-v1",
|
| 38 |
+
"train_count": 150,
|
| 39 |
+
"eval_count": 27
|
| 40 |
+
},
|
| 41 |
+
{
|
| 42 |
+
"name": "lighting",
|
| 43 |
+
"profiles": ["clean", "low_light", "uneven_light"],
|
| 44 |
+
"train_manifest": "data/ablation-limbu-namdhinggo-augmented-v1/train-lighting.jsonl",
|
| 45 |
+
"eval_manifest": "data/ablation-limbu-namdhinggo-augmented-v1/eval-lighting.jsonl",
|
| 46 |
+
"audit": "outputs/ablation-limbu-namdhinggo-augmented-v1/audit-lighting/rendered-degradation-split-audit.json",
|
| 47 |
+
"recipe_dir": "runs/sirijonga-limbu-paddle-ablation-lighting-v1",
|
| 48 |
+
"train_count": 450,
|
| 49 |
+
"eval_count": 81
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"name": "camera",
|
| 53 |
+
"profiles": ["clean", "camera_left", "camera_right", "phone_photo"],
|
| 54 |
+
"train_manifest": "data/ablation-limbu-namdhinggo-augmented-v1/train-camera.jsonl",
|
| 55 |
+
"eval_manifest": "data/ablation-limbu-namdhinggo-augmented-v1/eval-camera.jsonl",
|
| 56 |
+
"audit": "outputs/ablation-limbu-namdhinggo-augmented-v1/audit-camera/rendered-degradation-split-audit.json",
|
| 57 |
+
"recipe_dir": "runs/sirijonga-limbu-paddle-ablation-camera-v1",
|
| 58 |
+
"train_count": 600,
|
| 59 |
+
"eval_count": 108
|
| 60 |
+
}
|
| 61 |
+
],
|
| 62 |
+
"promotion_gates": [
|
| 63 |
+
"Every arm must pass rendered-degradation split audit with zero issues and warnings.",
|
| 64 |
+
"Every trained arm must be evaluated on the same held-out synthetic eval arm and on manually reviewed real Gorkhapatra/Naya Nepal Limbu references.",
|
| 65 |
+
"No synthetic-only result may be used as a paper accuracy claim.",
|
| 66 |
+
"A model can move toward engine integration only if it improves Limbu/Sirijonga real-reference OCR without regressing English, Nepali, table extraction, reading order, or figure/caption behavior in the broader document pipeline."
|
| 67 |
+
]
|
| 68 |
+
}
|
ocr-tech-mps-integration/configs/namdhinggo-font-assets-v1.json
ADDED
|
@@ -0,0 +1,106 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"manifest_id": "namdhinggo-font-assets-v1",
|
| 3 |
+
"description": "SIL Namdhinggo 3.100 font assets for Limbu/Sirijonga OCR synthesis. Assets are extracted from the upstream release ZIP and verified by package and member SHA-256.",
|
| 4 |
+
"assets": [
|
| 5 |
+
{
|
| 6 |
+
"id": "namdhinggo-regular",
|
| 7 |
+
"url": "https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip",
|
| 8 |
+
"archive_member": "Namdhinggo-3.100/Namdhinggo-Regular.ttf",
|
| 9 |
+
"archive_sha256": "073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915",
|
| 10 |
+
"target_path": "data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-Regular.ttf",
|
| 11 |
+
"sha256": "1b46b16277e4b6784bd19306a9474281888b656e9c57233b4e7dc63bd1229d6c",
|
| 12 |
+
"license": "OFL-1.1",
|
| 13 |
+
"source": "SIL font-namdhinggo release v3.100"
|
| 14 |
+
},
|
| 15 |
+
{
|
| 16 |
+
"id": "namdhinggo-medium",
|
| 17 |
+
"url": "https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip",
|
| 18 |
+
"archive_member": "Namdhinggo-3.100/Namdhinggo-Medium.ttf",
|
| 19 |
+
"archive_sha256": "073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915",
|
| 20 |
+
"target_path": "data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-Medium.ttf",
|
| 21 |
+
"sha256": "d29b8f9481dcf84567d2525a90561c8173375fbd1c761880b593c0956f3fdcb6",
|
| 22 |
+
"license": "OFL-1.1",
|
| 23 |
+
"source": "SIL font-namdhinggo release v3.100"
|
| 24 |
+
},
|
| 25 |
+
{
|
| 26 |
+
"id": "namdhinggo-semibold",
|
| 27 |
+
"url": "https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip",
|
| 28 |
+
"archive_member": "Namdhinggo-3.100/Namdhinggo-SemiBold.ttf",
|
| 29 |
+
"archive_sha256": "073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915",
|
| 30 |
+
"target_path": "data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-SemiBold.ttf",
|
| 31 |
+
"sha256": "f15fe69ee3c93e7f92609940ef17112e6d25807ca3df19cf81798928028c3397",
|
| 32 |
+
"license": "OFL-1.1",
|
| 33 |
+
"source": "SIL font-namdhinggo release v3.100"
|
| 34 |
+
},
|
| 35 |
+
{
|
| 36 |
+
"id": "namdhinggo-bold",
|
| 37 |
+
"url": "https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip",
|
| 38 |
+
"archive_member": "Namdhinggo-3.100/Namdhinggo-Bold.ttf",
|
| 39 |
+
"archive_sha256": "073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915",
|
| 40 |
+
"target_path": "data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-Bold.ttf",
|
| 41 |
+
"sha256": "76da18746fdbe1c6db4c1d0222a8f06dbc9d206e362ae06c928c6810123f57da",
|
| 42 |
+
"license": "OFL-1.1",
|
| 43 |
+
"source": "SIL font-namdhinggo release v3.100"
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"id": "namdhinggo-extrabold",
|
| 47 |
+
"url": "https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip",
|
| 48 |
+
"archive_member": "Namdhinggo-3.100/Namdhinggo-ExtraBold.ttf",
|
| 49 |
+
"archive_sha256": "073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915",
|
| 50 |
+
"target_path": "data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-ExtraBold.ttf",
|
| 51 |
+
"sha256": "ce3d01b4e99bcfe6a02941ac4c6f96f550b31a2af10e53454e1069e823bd9b2c",
|
| 52 |
+
"license": "OFL-1.1",
|
| 53 |
+
"source": "SIL font-namdhinggo release v3.100"
|
| 54 |
+
},
|
| 55 |
+
{
|
| 56 |
+
"id": "namdhinggo-ofl",
|
| 57 |
+
"url": "https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip",
|
| 58 |
+
"archive_member": "Namdhinggo-3.100/OFL.txt",
|
| 59 |
+
"archive_sha256": "073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915",
|
| 60 |
+
"target_path": "data/fonts/namdhinggo/Namdhinggo-3.100/OFL.txt",
|
| 61 |
+
"sha256": "64df695392d8facd1ed4b35e7158f9957073741c7f76c252c7423d5aae699cee",
|
| 62 |
+
"license": "OFL-1.1",
|
| 63 |
+
"source": "SIL font-namdhinggo release v3.100"
|
| 64 |
+
},
|
| 65 |
+
{
|
| 66 |
+
"id": "namdhinggo-ofl-faq",
|
| 67 |
+
"url": "https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip",
|
| 68 |
+
"archive_member": "Namdhinggo-3.100/OFL-FAQ.txt",
|
| 69 |
+
"archive_sha256": "073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915",
|
| 70 |
+
"target_path": "data/fonts/namdhinggo/Namdhinggo-3.100/OFL-FAQ.txt",
|
| 71 |
+
"sha256": "d6a2891c2470d349f5b3bb4d41c7901527c9a8016a29bfdd1cbec98b9420104e",
|
| 72 |
+
"license": "OFL-1.1",
|
| 73 |
+
"source": "SIL font-namdhinggo release v3.100"
|
| 74 |
+
},
|
| 75 |
+
{
|
| 76 |
+
"id": "namdhinggo-readme",
|
| 77 |
+
"url": "https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip",
|
| 78 |
+
"archive_member": "Namdhinggo-3.100/README.txt",
|
| 79 |
+
"archive_sha256": "073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915",
|
| 80 |
+
"target_path": "data/fonts/namdhinggo/Namdhinggo-3.100/README.txt",
|
| 81 |
+
"sha256": "99c8a1904df3f148628ce633a438011440114b912ddfa95b99c0132084466b64",
|
| 82 |
+
"license": "OFL-1.1",
|
| 83 |
+
"source": "SIL font-namdhinggo release v3.100"
|
| 84 |
+
},
|
| 85 |
+
{
|
| 86 |
+
"id": "namdhinggo-fontlog",
|
| 87 |
+
"url": "https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip",
|
| 88 |
+
"archive_member": "Namdhinggo-3.100/FONTLOG.txt",
|
| 89 |
+
"archive_sha256": "073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915",
|
| 90 |
+
"target_path": "data/fonts/namdhinggo/Namdhinggo-3.100/FONTLOG.txt",
|
| 91 |
+
"sha256": "505040639d06948d1a7ee154be618b55846e7561622c118af3fc62371508efa9",
|
| 92 |
+
"license": "OFL-1.1",
|
| 93 |
+
"source": "SIL font-namdhinggo release v3.100"
|
| 94 |
+
},
|
| 95 |
+
{
|
| 96 |
+
"id": "namdhinggo-fontmanifest",
|
| 97 |
+
"url": "https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip",
|
| 98 |
+
"archive_member": "Namdhinggo-3.100/fontmanifest.json",
|
| 99 |
+
"archive_sha256": "073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915",
|
| 100 |
+
"target_path": "data/fonts/namdhinggo/Namdhinggo-3.100/fontmanifest.json",
|
| 101 |
+
"sha256": "4113c4fc112949eb5c72de21d3a98a77f13b334fa2281e3b78a85d1abdbf3721",
|
| 102 |
+
"license": "OFL-1.1",
|
| 103 |
+
"source": "SIL font-namdhinggo release v3.100"
|
| 104 |
+
}
|
| 105 |
+
]
|
| 106 |
+
}
|
ocr-tech-mps-integration/configs/nepal-font-assets-v1.json
ADDED
|
@@ -0,0 +1,110 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"manifest_id": "nepal-font-assets-v1",
|
| 3 |
+
"description": "License-clear Unicode font assets used for Nepal multilingual OCR synthesis and renderability audits. Font binaries are provisioned into ignored data/fonts runtime data; this manifest records reproducible source URLs and SHA-256 checks.",
|
| 4 |
+
"assets": [
|
| 5 |
+
{
|
| 6 |
+
"id": "kanchenjunga-regular",
|
| 7 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/kanchenjunga/Kanchenjunga-Regular.ttf",
|
| 8 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/kanchenjunga/Kanchenjunga-Regular.ttf",
|
| 9 |
+
"sha256": "45609f8cc90d4733d3d1665346d359afc1a659def340d0f369121af034322ef9",
|
| 10 |
+
"license": "OFL-1.1",
|
| 11 |
+
"source": "Google Fonts ofl/kanchenjunga"
|
| 12 |
+
},
|
| 13 |
+
{
|
| 14 |
+
"id": "kanchenjunga-bold",
|
| 15 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/kanchenjunga/Kanchenjunga-Bold.ttf",
|
| 16 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/kanchenjunga/Kanchenjunga-Bold.ttf",
|
| 17 |
+
"sha256": "67a9a29cd49a17832a5eee8d2747fb9a66ee32e49dc52f5dd43887ff24eb0935",
|
| 18 |
+
"license": "OFL-1.1",
|
| 19 |
+
"source": "Google Fonts ofl/kanchenjunga"
|
| 20 |
+
},
|
| 21 |
+
{
|
| 22 |
+
"id": "kanchenjunga-ofl",
|
| 23 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/kanchenjunga/OFL.txt",
|
| 24 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/kanchenjunga/OFL.txt",
|
| 25 |
+
"sha256": "b6d9eb7d23db4cb33585dc14ae4cd4c4d4f90f0d20e1673197d92cb9af76283f",
|
| 26 |
+
"license": "OFL-1.1",
|
| 27 |
+
"source": "Google Fonts ofl/kanchenjunga"
|
| 28 |
+
},
|
| 29 |
+
{
|
| 30 |
+
"id": "kanchenjunga-upstream-info",
|
| 31 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/kanchenjunga/upstream_info.md",
|
| 32 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/kanchenjunga/upstream_info.md",
|
| 33 |
+
"sha256": "fc6cc71cf5168d56715f60fd77c79d5837e8777a416afd7f29382749f664c90d",
|
| 34 |
+
"license": "OFL-1.1",
|
| 35 |
+
"source": "Google Fonts ofl/kanchenjunga"
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"id": "noto-sans-tirhuta-regular",
|
| 39 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/notosanstirhuta/NotoSansTirhuta-Regular.ttf",
|
| 40 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/notosanstirhuta/NotoSansTirhuta-Regular.ttf",
|
| 41 |
+
"sha256": "ad7123ee63118b83ed2f723591e5e861baad8dd157508b8339362850c6036efe",
|
| 42 |
+
"license": "OFL-1.1",
|
| 43 |
+
"source": "Google Fonts ofl/notosanstirhuta"
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"id": "noto-sans-tirhuta-ofl",
|
| 47 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/notosanstirhuta/OFL.txt",
|
| 48 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/notosanstirhuta/OFL.txt",
|
| 49 |
+
"sha256": "9bf9c1ab9b7837bb65fb40dd7f6b706a28199cd633b80dcebe532bddc56594cb",
|
| 50 |
+
"license": "OFL-1.1",
|
| 51 |
+
"source": "Google Fonts ofl/notosanstirhuta"
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"id": "noto-sans-tirhuta-upstream-info",
|
| 55 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/notosanstirhuta/upstream_info.md",
|
| 56 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/notosanstirhuta/upstream_info.md",
|
| 57 |
+
"sha256": "5bedd6efb51804b87c4dc169e5aa912bb432027f2a5cd3b5a872f192e2464b59",
|
| 58 |
+
"license": "OFL-1.1",
|
| 59 |
+
"source": "Google Fonts ofl/notosanstirhuta"
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"id": "noto-serif-devanagari-variable",
|
| 63 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/notoserifdevanagari/NotoSerifDevanagari%5Bwdth%2Cwght%5D.ttf",
|
| 64 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/notoserifdevanagari/NotoSerifDevanagari-VariableFont_wdth,wght.ttf",
|
| 65 |
+
"sha256": "1191e07bfeb062d80e252eb85b0eafdfbda1e350707a2a60628668e8f677dbbb",
|
| 66 |
+
"license": "OFL-1.1",
|
| 67 |
+
"source": "Google Fonts ofl/notoserifdevanagari"
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"id": "noto-serif-devanagari-ofl",
|
| 71 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/notoserifdevanagari/OFL.txt",
|
| 72 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/notoserifdevanagari/OFL.txt",
|
| 73 |
+
"sha256": "a216f6f8d85c7228093e0ee5e258d9d377e6671f68acb4db1930b29583d0f331",
|
| 74 |
+
"license": "OFL-1.1",
|
| 75 |
+
"source": "Google Fonts ofl/notoserifdevanagari"
|
| 76 |
+
},
|
| 77 |
+
{
|
| 78 |
+
"id": "noto-serif-devanagari-upstream-info",
|
| 79 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/notoserifdevanagari/upstream_info.md",
|
| 80 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/notoserifdevanagari/upstream_info.md",
|
| 81 |
+
"sha256": "44dfc44c289514753ec0b3dd499bfbca110c907fd4dfc9c21cfb7716af3d94e3",
|
| 82 |
+
"license": "OFL-1.1",
|
| 83 |
+
"source": "Google Fonts ofl/notoserifdevanagari"
|
| 84 |
+
},
|
| 85 |
+
{
|
| 86 |
+
"id": "noto-sans-ol-chiki-variable",
|
| 87 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/notosansolchiki/NotoSansOlChiki%5Bwght%5D.ttf",
|
| 88 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/notosansolchiki/NotoSansOlChiki-VariableFont_wght.ttf",
|
| 89 |
+
"sha256": "c9c31988656f49eccec9588825ab3b5045099c2f850ef98f356f976e8a596b4d",
|
| 90 |
+
"license": "OFL-1.1",
|
| 91 |
+
"source": "Google Fonts ofl/notosansolchiki"
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"id": "noto-sans-ol-chiki-ofl",
|
| 95 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/notosansolchiki/OFL.txt",
|
| 96 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/notosansolchiki/OFL.txt",
|
| 97 |
+
"sha256": "b30754bbe6bea30b666a60d513ae80435a197979c9500e8380f83acd99dc763b",
|
| 98 |
+
"license": "OFL-1.1",
|
| 99 |
+
"source": "Google Fonts ofl/notosansolchiki"
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"id": "noto-sans-ol-chiki-upstream-info",
|
| 103 |
+
"url": "https://raw.githubusercontent.com/google/fonts/main/ofl/notosansolchiki/upstream_info.md",
|
| 104 |
+
"target_path": "data/fonts/google-fonts-2026-06-16/notosansolchiki/upstream_info.md",
|
| 105 |
+
"sha256": "16e693164bea4aa66cdf0cf4f64fa113f8b1722f001775c2b5a8a2853f50da6e",
|
| 106 |
+
"license": "OFL-1.1",
|
| 107 |
+
"source": "Google Fonts ofl/notosansolchiki"
|
| 108 |
+
}
|
| 109 |
+
]
|
| 110 |
+
}
|
ocr-tech-mps-integration/configs/nepal-language-pack-pipeline-v1.json
ADDED
|
@@ -0,0 +1,1051 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config_id": "nepal-language-pack-pipeline-v1",
|
| 3 |
+
"updated": "2026-07-04",
|
| 4 |
+
"description": "Language-pack-parameterized Paddle OCR pipeline registry: per-language script routing, converter, recognizer, and harvest state. Companion to RESEARCH.md sections 'The Paddle Pipeline' and 'Getting More Gorkhapatra Pages'.",
|
| 5 |
+
"pipeline_stages": [
|
| 6 |
+
"capture_prep",
|
| 7 |
+
"detection",
|
| 8 |
+
"script_routing",
|
| 9 |
+
"recognition",
|
| 10 |
+
"target_region_filtering",
|
| 11 |
+
"reading_order_assembly",
|
| 12 |
+
"export",
|
| 13 |
+
"eval_guardrail"
|
| 14 |
+
],
|
| 15 |
+
"harvest_sources": {
|
| 16 |
+
"audit": "outputs/gorkhapatra-language-pages-full-2026-07-04/gorkhapatra-language-pages.json",
|
| 17 |
+
"inventory": "outputs/language-pack-inventory-full-2026-07-04/language-pack-inventory.json"
|
| 18 |
+
},
|
| 19 |
+
"script_recognizers": {
|
| 20 |
+
"limbu_sirijonga": {
|
| 21 |
+
"status": "champion",
|
| 22 |
+
"artifact": "runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23",
|
| 23 |
+
"dictionary": "data/dictionaries/limbu-sirijonga-v2.txt",
|
| 24 |
+
"input_width": 480,
|
| 25 |
+
"eval": "grapheme CER 0.041 real Gorkhapatra (frozen silver eval cp 0.011)"
|
| 26 |
+
},
|
| 27 |
+
"devanagari": {
|
| 28 |
+
"status": "trained",
|
| 29 |
+
"artifact": "outputs/devanagari-recognizer-v2-2026-06-18",
|
| 30 |
+
"dictionary": "data/dictionaries/devanagari-v1.txt",
|
| 31 |
+
"eval": "grapheme CER 0.075-0.129 across 8 languages, real Gorkhapatra"
|
| 32 |
+
},
|
| 33 |
+
"sunuwar_jenticha": {
|
| 34 |
+
"status": "trained_remote_needs_export",
|
| 35 |
+
"artifact": "aws-ebs (same-font retrain 2026-06-19; not exported locally)",
|
| 36 |
+
"dictionary": "data/dictionaries/sunuwar-v1.txt",
|
| 37 |
+
"eval": "cp=graph CER 0.121 real Sikkim Herald crops"
|
| 38 |
+
},
|
| 39 |
+
"kiratrai": {
|
| 40 |
+
"status": "trained_weak_eval",
|
| 41 |
+
"artifact": "runs/kiratrai-ppocrv5-mobile",
|
| 42 |
+
"dictionary": "data/dictionaries/kiratrai-v1.txt",
|
| 43 |
+
"eval": "70 real crops only; converter coverage blocks expansion"
|
| 44 |
+
},
|
| 45 |
+
"newa_prachalit": {
|
| 46 |
+
"status": "trained_weak",
|
| 47 |
+
"artifact": "outputs/newa-recognizer-v3-2026-06-18",
|
| 48 |
+
"dictionary": "data/dictionaries/newa-native-v1.txt",
|
| 49 |
+
"eval": "grapheme CER 0.613 synthetic-only (honest fail)",
|
| 50 |
+
"focus_note": "Newar is focus language 2 (steer 2026-07-04); Devanagari route first, native Prachalit needs real printed-register data"
|
| 51 |
+
},
|
| 52 |
+
"tirhuta": {
|
| 53 |
+
"status": "trained_weak",
|
| 54 |
+
"artifact": "outputs/tirhuta-recognizer-v1-2026-06-18",
|
| 55 |
+
"dictionary": "data/dictionaries/tirhuta-v1.txt",
|
| 56 |
+
"eval": "grapheme CER 0.343 synthetic-only"
|
| 57 |
+
},
|
| 58 |
+
"lepcha": {
|
| 59 |
+
"status": "data_only",
|
| 60 |
+
"artifact": null,
|
| 61 |
+
"dictionary": "data/dictionaries/lepcha-v1.txt",
|
| 62 |
+
"eval": "132 real crops; converter blocked outside eval editions"
|
| 63 |
+
},
|
| 64 |
+
"gurung_khema": {
|
| 65 |
+
"status": "data_only_synthetic_by_nature",
|
| 66 |
+
"artifact": null,
|
| 67 |
+
"dictionary": "data/dictionaries/gurungkhema-v1.txt",
|
| 68 |
+
"eval": "no real print exists (revival script)"
|
| 69 |
+
},
|
| 70 |
+
"tibetan": {
|
| 71 |
+
"status": "unstarted",
|
| 72 |
+
"artifact": null,
|
| 73 |
+
"dictionary": null,
|
| 74 |
+
"eval": "Gorkhapatra Tibetan-script pages are visual-only (verified 2026-07-04)"
|
| 75 |
+
},
|
| 76 |
+
"ol_chiki": {
|
| 77 |
+
"status": "unstarted",
|
| 78 |
+
"artifact": null,
|
| 79 |
+
"dictionary": null,
|
| 80 |
+
"eval": null
|
| 81 |
+
},
|
| 82 |
+
"devanagari_limbu": {
|
| 83 |
+
"status": "trained",
|
| 84 |
+
"artifact": "runs/limbu-devanagari-cdc-ppocrv5-mobile-training-bundle-2026-06-21",
|
| 85 |
+
"dictionary": "data/dictionaries/devanagari-v1.txt",
|
| 86 |
+
"eval": "CDC Devanagari-Limbu book path"
|
| 87 |
+
},
|
| 88 |
+
"perso_arabic": {
|
| 89 |
+
"status": "unstarted",
|
| 90 |
+
"artifact": null,
|
| 91 |
+
"dictionary": null,
|
| 92 |
+
"eval": "Urdu pages; script/text-layer class unverified"
|
| 93 |
+
}
|
| 94 |
+
},
|
| 95 |
+
"languages": [
|
| 96 |
+
{
|
| 97 |
+
"label": "खश जुम्ली",
|
| 98 |
+
"aliases_seen": [
|
| 99 |
+
"खश जुम्ली"
|
| 100 |
+
],
|
| 101 |
+
"registry_id": null,
|
| 102 |
+
"primary_script": "devanagari",
|
| 103 |
+
"secondary_script": null,
|
| 104 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 105 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 106 |
+
"recognizer_ref": "devanagari",
|
| 107 |
+
"harvest_2026_07_04": {
|
| 108 |
+
"items": 9,
|
| 109 |
+
"aligned": 6,
|
| 110 |
+
"page_hits": 13,
|
| 111 |
+
"unique_dates": 9
|
| 112 |
+
},
|
| 113 |
+
"status": "harvested_pending_silver_pack"
|
| 114 |
+
},
|
| 115 |
+
{
|
| 116 |
+
"label": "अवधी",
|
| 117 |
+
"aliases_seen": [
|
| 118 |
+
"अवधी"
|
| 119 |
+
],
|
| 120 |
+
"registry_id": null,
|
| 121 |
+
"primary_script": "devanagari",
|
| 122 |
+
"secondary_script": null,
|
| 123 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 124 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 125 |
+
"recognizer_ref": "devanagari",
|
| 126 |
+
"harvest_2026_07_04": {
|
| 127 |
+
"items": 9,
|
| 128 |
+
"aligned": 5,
|
| 129 |
+
"page_hits": 12,
|
| 130 |
+
"unique_dates": 9
|
| 131 |
+
},
|
| 132 |
+
"status": "harvested_pending_silver_pack"
|
| 133 |
+
},
|
| 134 |
+
{
|
| 135 |
+
"label": "थामी",
|
| 136 |
+
"aliases_seen": [
|
| 137 |
+
"थामी"
|
| 138 |
+
],
|
| 139 |
+
"registry_id": null,
|
| 140 |
+
"primary_script": "devanagari",
|
| 141 |
+
"secondary_script": null,
|
| 142 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 143 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 144 |
+
"recognizer_ref": "devanagari",
|
| 145 |
+
"harvest_2026_07_04": {
|
| 146 |
+
"items": 9,
|
| 147 |
+
"aligned": 6,
|
| 148 |
+
"page_hits": 12,
|
| 149 |
+
"unique_dates": 9
|
| 150 |
+
},
|
| 151 |
+
"status": "harvested_pending_silver_pack"
|
| 152 |
+
},
|
| 153 |
+
{
|
| 154 |
+
"label": "राजवंशी",
|
| 155 |
+
"aliases_seen": [
|
| 156 |
+
"राजवंशी"
|
| 157 |
+
],
|
| 158 |
+
"registry_id": "rajbanshi",
|
| 159 |
+
"primary_script": "devanagari",
|
| 160 |
+
"secondary_script": null,
|
| 161 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 162 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 163 |
+
"recognizer_ref": "devanagari",
|
| 164 |
+
"harvest_2026_07_04": {
|
| 165 |
+
"items": 9,
|
| 166 |
+
"aligned": 6,
|
| 167 |
+
"page_hits": 12,
|
| 168 |
+
"unique_dates": 9
|
| 169 |
+
},
|
| 170 |
+
"status": "harvested_pending_silver_pack"
|
| 171 |
+
},
|
| 172 |
+
{
|
| 173 |
+
"label": "ल्होमी",
|
| 174 |
+
"aliases_seen": [
|
| 175 |
+
"ल्होमी"
|
| 176 |
+
],
|
| 177 |
+
"registry_id": null,
|
| 178 |
+
"primary_script": "devanagari",
|
| 179 |
+
"secondary_script": null,
|
| 180 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 181 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 182 |
+
"recognizer_ref": "devanagari",
|
| 183 |
+
"harvest_2026_07_04": {
|
| 184 |
+
"items": 9,
|
| 185 |
+
"aligned": 6,
|
| 186 |
+
"page_hits": 12,
|
| 187 |
+
"unique_dates": 9
|
| 188 |
+
},
|
| 189 |
+
"status": "harvested_pending_silver_pack"
|
| 190 |
+
},
|
| 191 |
+
{
|
| 192 |
+
"label": "ताजपुरिया",
|
| 193 |
+
"aliases_seen": [
|
| 194 |
+
"ताजपुरिया"
|
| 195 |
+
],
|
| 196 |
+
"registry_id": null,
|
| 197 |
+
"primary_script": "devanagari",
|
| 198 |
+
"secondary_script": null,
|
| 199 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 200 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 201 |
+
"recognizer_ref": "devanagari",
|
| 202 |
+
"harvest_2026_07_04": {
|
| 203 |
+
"items": 9,
|
| 204 |
+
"aligned": 5,
|
| 205 |
+
"page_hits": 11,
|
| 206 |
+
"unique_dates": 9
|
| 207 |
+
},
|
| 208 |
+
"status": "harvested_pending_silver_pack"
|
| 209 |
+
},
|
| 210 |
+
{
|
| 211 |
+
"label": "किसान",
|
| 212 |
+
"aliases_seen": [
|
| 213 |
+
"किसान"
|
| 214 |
+
],
|
| 215 |
+
"registry_id": null,
|
| 216 |
+
"primary_script": "devanagari",
|
| 217 |
+
"secondary_script": null,
|
| 218 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 219 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 220 |
+
"recognizer_ref": "devanagari",
|
| 221 |
+
"harvest_2026_07_04": {
|
| 222 |
+
"items": 10,
|
| 223 |
+
"aligned": 5,
|
| 224 |
+
"page_hits": 10,
|
| 225 |
+
"unique_dates": 10
|
| 226 |
+
},
|
| 227 |
+
"status": "harvested_pending_silver_pack"
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"label": "माझी",
|
| 231 |
+
"aliases_seen": [
|
| 232 |
+
"माझी"
|
| 233 |
+
],
|
| 234 |
+
"registry_id": null,
|
| 235 |
+
"primary_script": "devanagari",
|
| 236 |
+
"secondary_script": null,
|
| 237 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 238 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 239 |
+
"recognizer_ref": "devanagari",
|
| 240 |
+
"harvest_2026_07_04": {
|
| 241 |
+
"items": 10,
|
| 242 |
+
"aligned": 5,
|
| 243 |
+
"page_hits": 10,
|
| 244 |
+
"unique_dates": 10
|
| 245 |
+
},
|
| 246 |
+
"status": "harvested_pending_silver_pack"
|
| 247 |
+
},
|
| 248 |
+
{
|
| 249 |
+
"label": "सुनुवार",
|
| 250 |
+
"aliases_seen": [
|
| 251 |
+
"सुनुवार"
|
| 252 |
+
],
|
| 253 |
+
"registry_id": "sunuwar",
|
| 254 |
+
"primary_script": "devanagari",
|
| 255 |
+
"secondary_script": "sunuwar_jenticha",
|
| 256 |
+
"text_layer_class": "legacy_font_nayanepal_devanagari (verified render 2082-10-16; native Jenticha chart box present)",
|
| 257 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal')",
|
| 258 |
+
"recognizer_ref": "devanagari",
|
| 259 |
+
"harvest_2026_07_04": {
|
| 260 |
+
"items": 10,
|
| 261 |
+
"aligned": 5,
|
| 262 |
+
"page_hits": 10,
|
| 263 |
+
"unique_dates": 10
|
| 264 |
+
},
|
| 265 |
+
"status": "harvested_pending_silver_pack"
|
| 266 |
+
},
|
| 267 |
+
{
|
| 268 |
+
"label": "कुमाल",
|
| 269 |
+
"aliases_seen": [
|
| 270 |
+
"कुमाल"
|
| 271 |
+
],
|
| 272 |
+
"registry_id": null,
|
| 273 |
+
"primary_script": "devanagari",
|
| 274 |
+
"secondary_script": null,
|
| 275 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 276 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 277 |
+
"recognizer_ref": "devanagari",
|
| 278 |
+
"harvest_2026_07_04": {
|
| 279 |
+
"items": 9,
|
| 280 |
+
"aligned": 5,
|
| 281 |
+
"page_hits": 10,
|
| 282 |
+
"unique_dates": 9
|
| 283 |
+
},
|
| 284 |
+
"status": "harvested_pending_silver_pack"
|
| 285 |
+
},
|
| 286 |
+
{
|
| 287 |
+
"label": "थारु",
|
| 288 |
+
"aliases_seen": [
|
| 289 |
+
"थारु",
|
| 290 |
+
"थारू"
|
| 291 |
+
],
|
| 292 |
+
"registry_id": "rana-tharu",
|
| 293 |
+
"primary_script": "devanagari",
|
| 294 |
+
"secondary_script": null,
|
| 295 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 296 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 297 |
+
"recognizer_ref": "devanagari",
|
| 298 |
+
"harvest_2026_07_04": {
|
| 299 |
+
"items": 9,
|
| 300 |
+
"aligned": 5,
|
| 301 |
+
"page_hits": 10,
|
| 302 |
+
"unique_dates": 9
|
| 303 |
+
},
|
| 304 |
+
"status": "harvested_pending_silver_pack"
|
| 305 |
+
},
|
| 306 |
+
{
|
| 307 |
+
"label": "बज्जिका",
|
| 308 |
+
"aliases_seen": [
|
| 309 |
+
"बज्जिका"
|
| 310 |
+
],
|
| 311 |
+
"registry_id": null,
|
| 312 |
+
"primary_script": "devanagari",
|
| 313 |
+
"secondary_script": null,
|
| 314 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 315 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 316 |
+
"recognizer_ref": "devanagari",
|
| 317 |
+
"harvest_2026_07_04": {
|
| 318 |
+
"items": 9,
|
| 319 |
+
"aligned": 5,
|
| 320 |
+
"page_hits": 10,
|
| 321 |
+
"unique_dates": 9
|
| 322 |
+
},
|
| 323 |
+
"status": "harvested_pending_silver_pack"
|
| 324 |
+
},
|
| 325 |
+
{
|
| 326 |
+
"label": "बान्तावा",
|
| 327 |
+
"aliases_seen": [
|
| 328 |
+
"बान्तावा"
|
| 329 |
+
],
|
| 330 |
+
"registry_id": "bantawa",
|
| 331 |
+
"primary_script": "devanagari",
|
| 332 |
+
"secondary_script": null,
|
| 333 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 334 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 335 |
+
"recognizer_ref": "devanagari",
|
| 336 |
+
"harvest_2026_07_04": {
|
| 337 |
+
"items": 9,
|
| 338 |
+
"aligned": 5,
|
| 339 |
+
"page_hits": 10,
|
| 340 |
+
"unique_dates": 9
|
| 341 |
+
},
|
| 342 |
+
"status": "harvested_pending_silver_pack"
|
| 343 |
+
},
|
| 344 |
+
{
|
| 345 |
+
"label": "बाहिङ",
|
| 346 |
+
"aliases_seen": [
|
| 347 |
+
"बाहिङ"
|
| 348 |
+
],
|
| 349 |
+
"registry_id": null,
|
| 350 |
+
"primary_script": "devanagari",
|
| 351 |
+
"secondary_script": null,
|
| 352 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 353 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 354 |
+
"recognizer_ref": "devanagari",
|
| 355 |
+
"harvest_2026_07_04": {
|
| 356 |
+
"items": 9,
|
| 357 |
+
"aligned": 5,
|
| 358 |
+
"page_hits": 10,
|
| 359 |
+
"unique_dates": 9
|
| 360 |
+
},
|
| 361 |
+
"status": "harvested_pending_silver_pack"
|
| 362 |
+
},
|
| 363 |
+
{
|
| 364 |
+
"label": "मुगाल",
|
| 365 |
+
"aliases_seen": [
|
| 366 |
+
"मुगाल"
|
| 367 |
+
],
|
| 368 |
+
"registry_id": null,
|
| 369 |
+
"primary_script": "tibetan",
|
| 370 |
+
"secondary_script": null,
|
| 371 |
+
"text_layer_class": "tibetan_script_visual_only (verified sample 2026-06-24/0012)",
|
| 372 |
+
"converter": null,
|
| 373 |
+
"recognizer_ref": "tibetan",
|
| 374 |
+
"harvest_2026_07_04": {
|
| 375 |
+
"items": 9,
|
| 376 |
+
"aligned": 5,
|
| 377 |
+
"page_hits": 10,
|
| 378 |
+
"unique_dates": 9
|
| 379 |
+
},
|
| 380 |
+
"status": "harvested_pending_silver_pack"
|
| 381 |
+
},
|
| 382 |
+
{
|
| 383 |
+
"label": "ह्योल्मो",
|
| 384 |
+
"aliases_seen": [
|
| 385 |
+
"ह्योलमो",
|
| 386 |
+
"ह्योल्मो"
|
| 387 |
+
],
|
| 388 |
+
"registry_id": null,
|
| 389 |
+
"primary_script": "devanagari",
|
| 390 |
+
"secondary_script": null,
|
| 391 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 392 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 393 |
+
"recognizer_ref": "devanagari",
|
| 394 |
+
"harvest_2026_07_04": {
|
| 395 |
+
"items": 9,
|
| 396 |
+
"aligned": 5,
|
| 397 |
+
"page_hits": 10,
|
| 398 |
+
"unique_dates": 9
|
| 399 |
+
},
|
| 400 |
+
"status": "harvested_pending_silver_pack"
|
| 401 |
+
},
|
| 402 |
+
{
|
| 403 |
+
"label": "मारवाडी",
|
| 404 |
+
"aliases_seen": [
|
| 405 |
+
"मारवाडी"
|
| 406 |
+
],
|
| 407 |
+
"registry_id": null,
|
| 408 |
+
"primary_script": "devanagari",
|
| 409 |
+
"secondary_script": null,
|
| 410 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 411 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 412 |
+
"recognizer_ref": "devanagari",
|
| 413 |
+
"harvest_2026_07_04": {
|
| 414 |
+
"items": 10,
|
| 415 |
+
"aligned": 5,
|
| 416 |
+
"page_hits": 9,
|
| 417 |
+
"unique_dates": 10
|
| 418 |
+
},
|
| 419 |
+
"status": "harvested_pending_silver_pack"
|
| 420 |
+
},
|
| 421 |
+
{
|
| 422 |
+
"label": "दनुवार",
|
| 423 |
+
"aliases_seen": [
|
| 424 |
+
"दनुवार"
|
| 425 |
+
],
|
| 426 |
+
"registry_id": null,
|
| 427 |
+
"primary_script": "devanagari",
|
| 428 |
+
"secondary_script": null,
|
| 429 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 430 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 431 |
+
"recognizer_ref": "devanagari",
|
| 432 |
+
"harvest_2026_07_04": {
|
| 433 |
+
"items": 9,
|
| 434 |
+
"aligned": 5,
|
| 435 |
+
"page_hits": 9,
|
| 436 |
+
"unique_dates": 9
|
| 437 |
+
},
|
| 438 |
+
"status": "harvested_pending_silver_pack"
|
| 439 |
+
},
|
| 440 |
+
{
|
| 441 |
+
"label": "धिमाल",
|
| 442 |
+
"aliases_seen": [
|
| 443 |
+
"धिमाल"
|
| 444 |
+
],
|
| 445 |
+
"registry_id": "dhimal",
|
| 446 |
+
"primary_script": "devanagari",
|
| 447 |
+
"secondary_script": null,
|
| 448 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 449 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 450 |
+
"recognizer_ref": "devanagari",
|
| 451 |
+
"harvest_2026_07_04": {
|
| 452 |
+
"items": 9,
|
| 453 |
+
"aligned": 4,
|
| 454 |
+
"page_hits": 9,
|
| 455 |
+
"unique_dates": 9
|
| 456 |
+
},
|
| 457 |
+
"status": "harvested_pending_silver_pack"
|
| 458 |
+
},
|
| 459 |
+
{
|
| 460 |
+
"label": "भोजपुरी",
|
| 461 |
+
"aliases_seen": [
|
| 462 |
+
"भोजपुरी"
|
| 463 |
+
],
|
| 464 |
+
"registry_id": "bhojpuri",
|
| 465 |
+
"primary_script": "devanagari",
|
| 466 |
+
"secondary_script": null,
|
| 467 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 468 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 469 |
+
"recognizer_ref": "devanagari",
|
| 470 |
+
"harvest_2026_07_04": {
|
| 471 |
+
"items": 10,
|
| 472 |
+
"aligned": 4,
|
| 473 |
+
"page_hits": 8,
|
| 474 |
+
"unique_dates": 10
|
| 475 |
+
},
|
| 476 |
+
"status": "harvested_pending_silver_pack"
|
| 477 |
+
},
|
| 478 |
+
{
|
| 479 |
+
"label": "उराँव",
|
| 480 |
+
"aliases_seen": [
|
| 481 |
+
"उराँव"
|
| 482 |
+
],
|
| 483 |
+
"registry_id": null,
|
| 484 |
+
"primary_script": "devanagari",
|
| 485 |
+
"secondary_script": null,
|
| 486 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 487 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 488 |
+
"recognizer_ref": "devanagari",
|
| 489 |
+
"harvest_2026_07_04": {
|
| 490 |
+
"items": 9,
|
| 491 |
+
"aligned": 4,
|
| 492 |
+
"page_hits": 8,
|
| 493 |
+
"unique_dates": 9
|
| 494 |
+
},
|
| 495 |
+
"status": "harvested_pending_silver_pack"
|
| 496 |
+
},
|
| 497 |
+
{
|
| 498 |
+
"label": "लिम्बु",
|
| 499 |
+
"aliases_seen": [
|
| 500 |
+
"लिम्बू"
|
| 501 |
+
],
|
| 502 |
+
"registry_id": "limbu",
|
| 503 |
+
"primary_script": "limbu_sirijonga",
|
| 504 |
+
"secondary_script": "devanagari_limbu",
|
| 505 |
+
"text_layer_class": "legacy_font_namdhinggo (verified)",
|
| 506 |
+
"converter": "nepal_ttf2utf limbu / src/ocrtech/limbu_legacy.py + data/mappings/limbu-legacy/Limbu.map",
|
| 507 |
+
"recognizer_ref": "limbu_sirijonga",
|
| 508 |
+
"harvest_2026_07_04": {
|
| 509 |
+
"items": 9,
|
| 510 |
+
"aligned": 4,
|
| 511 |
+
"page_hits": 8,
|
| 512 |
+
"unique_dates": 9
|
| 513 |
+
},
|
| 514 |
+
"status": "harvested_pending_silver_pack"
|
| 515 |
+
},
|
| 516 |
+
{
|
| 517 |
+
"label": "गुरुङ",
|
| 518 |
+
"aliases_seen": [
|
| 519 |
+
"गुरुङ"
|
| 520 |
+
],
|
| 521 |
+
"registry_id": "gurung",
|
| 522 |
+
"primary_script": "devanagari",
|
| 523 |
+
"secondary_script": null,
|
| 524 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 525 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 526 |
+
"recognizer_ref": "devanagari",
|
| 527 |
+
"harvest_2026_07_04": {
|
| 528 |
+
"items": 8,
|
| 529 |
+
"aligned": 4,
|
| 530 |
+
"page_hits": 8,
|
| 531 |
+
"unique_dates": 8
|
| 532 |
+
},
|
| 533 |
+
"status": "harvested_pending_silver_pack"
|
| 534 |
+
},
|
| 535 |
+
{
|
| 536 |
+
"label": "तामाङ",
|
| 537 |
+
"aliases_seen": [
|
| 538 |
+
"तामाङ"
|
| 539 |
+
],
|
| 540 |
+
"registry_id": "tamang",
|
| 541 |
+
"primary_script": "devanagari",
|
| 542 |
+
"secondary_script": null,
|
| 543 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 544 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 545 |
+
"recognizer_ref": "devanagari",
|
| 546 |
+
"harvest_2026_07_04": {
|
| 547 |
+
"items": 8,
|
| 548 |
+
"aligned": 4,
|
| 549 |
+
"page_hits": 8,
|
| 550 |
+
"unique_dates": 8
|
| 551 |
+
},
|
| 552 |
+
"status": "harvested_pending_silver_pack"
|
| 553 |
+
},
|
| 554 |
+
{
|
| 555 |
+
"label": "मगही",
|
| 556 |
+
"aliases_seen": [
|
| 557 |
+
"मगही"
|
| 558 |
+
],
|
| 559 |
+
"registry_id": null,
|
| 560 |
+
"primary_script": "devanagari",
|
| 561 |
+
"secondary_script": null,
|
| 562 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 563 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 564 |
+
"recognizer_ref": "devanagari",
|
| 565 |
+
"harvest_2026_07_04": {
|
| 566 |
+
"items": 8,
|
| 567 |
+
"aligned": 4,
|
| 568 |
+
"page_hits": 8,
|
| 569 |
+
"unique_dates": 8
|
| 570 |
+
},
|
| 571 |
+
"status": "harvested_pending_silver_pack"
|
| 572 |
+
},
|
| 573 |
+
{
|
| 574 |
+
"label": "वाम्बुले",
|
| 575 |
+
"aliases_seen": [
|
| 576 |
+
"वाम्बुले"
|
| 577 |
+
],
|
| 578 |
+
"registry_id": "wambule",
|
| 579 |
+
"primary_script": "devanagari",
|
| 580 |
+
"secondary_script": null,
|
| 581 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 582 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 583 |
+
"recognizer_ref": "devanagari",
|
| 584 |
+
"harvest_2026_07_04": {
|
| 585 |
+
"items": 8,
|
| 586 |
+
"aligned": 4,
|
| 587 |
+
"page_hits": 8,
|
| 588 |
+
"unique_dates": 8
|
| 589 |
+
},
|
| 590 |
+
"status": "harvested_pending_silver_pack"
|
| 591 |
+
},
|
| 592 |
+
{
|
| 593 |
+
"label": "कुलुङ",
|
| 594 |
+
"aliases_seen": [
|
| 595 |
+
"कुलुङ"
|
| 596 |
+
],
|
| 597 |
+
"registry_id": "kulung",
|
| 598 |
+
"primary_script": "devanagari",
|
| 599 |
+
"secondary_script": null,
|
| 600 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 601 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 602 |
+
"recognizer_ref": "devanagari",
|
| 603 |
+
"harvest_2026_07_04": {
|
| 604 |
+
"items": 7,
|
| 605 |
+
"aligned": 4,
|
| 606 |
+
"page_hits": 8,
|
| 607 |
+
"unique_dates": 7
|
| 608 |
+
},
|
| 609 |
+
"status": "harvested_pending_silver_pack"
|
| 610 |
+
},
|
| 611 |
+
{
|
| 612 |
+
"label": "राना थारु",
|
| 613 |
+
"aliases_seen": [
|
| 614 |
+
"राना थारु"
|
| 615 |
+
],
|
| 616 |
+
"registry_id": "rana-tharu",
|
| 617 |
+
"primary_script": "devanagari",
|
| 618 |
+
"secondary_script": null,
|
| 619 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 620 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 621 |
+
"recognizer_ref": "devanagari",
|
| 622 |
+
"harvest_2026_07_04": {
|
| 623 |
+
"items": 7,
|
| 624 |
+
"aligned": 4,
|
| 625 |
+
"page_hits": 8,
|
| 626 |
+
"unique_dates": 7
|
| 627 |
+
},
|
| 628 |
+
"status": "harvested_pending_silver_pack"
|
| 629 |
+
},
|
| 630 |
+
{
|
| 631 |
+
"label": "चेपाङ",
|
| 632 |
+
"aliases_seen": [
|
| 633 |
+
"चेपाङ"
|
| 634 |
+
],
|
| 635 |
+
"registry_id": null,
|
| 636 |
+
"primary_script": "devanagari",
|
| 637 |
+
"secondary_script": null,
|
| 638 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 639 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 640 |
+
"recognizer_ref": "devanagari",
|
| 641 |
+
"harvest_2026_07_04": {
|
| 642 |
+
"items": 8,
|
| 643 |
+
"aligned": 3,
|
| 644 |
+
"page_hits": 7,
|
| 645 |
+
"unique_dates": 8
|
| 646 |
+
},
|
| 647 |
+
"status": "harvested_pending_silver_pack"
|
| 648 |
+
},
|
| 649 |
+
{
|
| 650 |
+
"label": "बराम",
|
| 651 |
+
"aliases_seen": [
|
| 652 |
+
"बराम"
|
| 653 |
+
],
|
| 654 |
+
"registry_id": null,
|
| 655 |
+
"primary_script": "devanagari",
|
| 656 |
+
"secondary_script": null,
|
| 657 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 658 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 659 |
+
"recognizer_ref": "devanagari",
|
| 660 |
+
"harvest_2026_07_04": {
|
| 661 |
+
"items": 10,
|
| 662 |
+
"aligned": 3,
|
| 663 |
+
"page_hits": 6,
|
| 664 |
+
"unique_dates": 10
|
| 665 |
+
},
|
| 666 |
+
"status": "harvested_pending_silver_pack"
|
| 667 |
+
},
|
| 668 |
+
{
|
| 669 |
+
"label": "चाम्लिङ राई",
|
| 670 |
+
"aliases_seen": [
|
| 671 |
+
"चाम्लिङ",
|
| 672 |
+
"चाम्लिङ राई"
|
| 673 |
+
],
|
| 674 |
+
"registry_id": null,
|
| 675 |
+
"primary_script": "devanagari",
|
| 676 |
+
"secondary_script": null,
|
| 677 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 678 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 679 |
+
"recognizer_ref": "devanagari",
|
| 680 |
+
"harvest_2026_07_04": {
|
| 681 |
+
"items": 9,
|
| 682 |
+
"aligned": 3,
|
| 683 |
+
"page_hits": 6,
|
| 684 |
+
"unique_dates": 9
|
| 685 |
+
},
|
| 686 |
+
"status": "harvested_pending_silver_pack"
|
| 687 |
+
},
|
| 688 |
+
{
|
| 689 |
+
"label": "डोट्याली",
|
| 690 |
+
"aliases_seen": [
|
| 691 |
+
"डोट्याली"
|
| 692 |
+
],
|
| 693 |
+
"registry_id": null,
|
| 694 |
+
"primary_script": "devanagari",
|
| 695 |
+
"secondary_script": null,
|
| 696 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 697 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 698 |
+
"recognizer_ref": "devanagari",
|
| 699 |
+
"harvest_2026_07_04": {
|
| 700 |
+
"items": 9,
|
| 701 |
+
"aligned": 3,
|
| 702 |
+
"page_hits": 6,
|
| 703 |
+
"unique_dates": 9
|
| 704 |
+
},
|
| 705 |
+
"status": "harvested_pending_silver_pack"
|
| 706 |
+
},
|
| 707 |
+
{
|
| 708 |
+
"label": "बझाङी",
|
| 709 |
+
"aliases_seen": [
|
| 710 |
+
"बझाङी"
|
| 711 |
+
],
|
| 712 |
+
"registry_id": null,
|
| 713 |
+
"primary_script": "devanagari",
|
| 714 |
+
"secondary_script": null,
|
| 715 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 716 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 717 |
+
"recognizer_ref": "devanagari",
|
| 718 |
+
"harvest_2026_07_04": {
|
| 719 |
+
"items": 9,
|
| 720 |
+
"aligned": 3,
|
| 721 |
+
"page_hits": 6,
|
| 722 |
+
"unique_dates": 9
|
| 723 |
+
},
|
| 724 |
+
"status": "harvested_pending_silver_pack"
|
| 725 |
+
},
|
| 726 |
+
{
|
| 727 |
+
"label": "मगर",
|
| 728 |
+
"aliases_seen": [
|
| 729 |
+
"मगर"
|
| 730 |
+
],
|
| 731 |
+
"registry_id": "magar",
|
| 732 |
+
"primary_script": "devanagari",
|
| 733 |
+
"secondary_script": null,
|
| 734 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 735 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 736 |
+
"recognizer_ref": "devanagari",
|
| 737 |
+
"harvest_2026_07_04": {
|
| 738 |
+
"items": 9,
|
| 739 |
+
"aligned": 3,
|
| 740 |
+
"page_hits": 6,
|
| 741 |
+
"unique_dates": 9
|
| 742 |
+
},
|
| 743 |
+
"status": "harvested_pending_silver_pack"
|
| 744 |
+
},
|
| 745 |
+
{
|
| 746 |
+
"label": "अछामि",
|
| 747 |
+
"aliases_seen": [
|
| 748 |
+
"अछामि"
|
| 749 |
+
],
|
| 750 |
+
"registry_id": null,
|
| 751 |
+
"primary_script": "devanagari",
|
| 752 |
+
"secondary_script": null,
|
| 753 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 754 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 755 |
+
"recognizer_ref": "devanagari",
|
| 756 |
+
"harvest_2026_07_04": {
|
| 757 |
+
"items": 8,
|
| 758 |
+
"aligned": 4,
|
| 759 |
+
"page_hits": 6,
|
| 760 |
+
"unique_dates": 8
|
| 761 |
+
},
|
| 762 |
+
"status": "harvested_pending_silver_pack"
|
| 763 |
+
},
|
| 764 |
+
{
|
| 765 |
+
"label": "जिरेल",
|
| 766 |
+
"aliases_seen": [
|
| 767 |
+
"जिरेल"
|
| 768 |
+
],
|
| 769 |
+
"registry_id": null,
|
| 770 |
+
"primary_script": "devanagari",
|
| 771 |
+
"secondary_script": null,
|
| 772 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 773 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 774 |
+
"recognizer_ref": "devanagari",
|
| 775 |
+
"harvest_2026_07_04": {
|
| 776 |
+
"items": 8,
|
| 777 |
+
"aligned": 3,
|
| 778 |
+
"page_hits": 6,
|
| 779 |
+
"unique_dates": 8
|
| 780 |
+
},
|
| 781 |
+
"status": "harvested_pending_silver_pack"
|
| 782 |
+
},
|
| 783 |
+
{
|
| 784 |
+
"label": "दराई",
|
| 785 |
+
"aliases_seen": [
|
| 786 |
+
"दराई"
|
| 787 |
+
],
|
| 788 |
+
"registry_id": null,
|
| 789 |
+
"primary_script": "devanagari",
|
| 790 |
+
"secondary_script": null,
|
| 791 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 792 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 793 |
+
"recognizer_ref": "devanagari",
|
| 794 |
+
"harvest_2026_07_04": {
|
| 795 |
+
"items": 8,
|
| 796 |
+
"aligned": 3,
|
| 797 |
+
"page_hits": 6,
|
| 798 |
+
"unique_dates": 8
|
| 799 |
+
},
|
| 800 |
+
"status": "harvested_pending_silver_pack"
|
| 801 |
+
},
|
| 802 |
+
{
|
| 803 |
+
"label": "नेपाल",
|
| 804 |
+
"aliases_seen": [
|
| 805 |
+
"नेपाल"
|
| 806 |
+
],
|
| 807 |
+
"registry_id": "newar",
|
| 808 |
+
"primary_script": "devanagari",
|
| 809 |
+
"secondary_script": "newa_prachalit",
|
| 810 |
+
"text_layer_class": "legacy_font_nayanepal_devanagari (verified via silver run 2026-07-04: 6,800 kept lines)",
|
| 811 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 812 |
+
"recognizer_ref": "devanagari",
|
| 813 |
+
"harvest_2026_07_04": {
|
| 814 |
+
"items": 8,
|
| 815 |
+
"aligned": 3,
|
| 816 |
+
"page_hits": 6,
|
| 817 |
+
"unique_dates": 8
|
| 818 |
+
},
|
| 819 |
+
"status": "focus_language_2_pending_frozen_split"
|
| 820 |
+
},
|
| 821 |
+
{
|
| 822 |
+
"label": "भोटे",
|
| 823 |
+
"aliases_seen": [
|
| 824 |
+
"भोटे"
|
| 825 |
+
],
|
| 826 |
+
"registry_id": null,
|
| 827 |
+
"primary_script": "devanagari",
|
| 828 |
+
"secondary_script": null,
|
| 829 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 830 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 831 |
+
"recognizer_ref": "devanagari",
|
| 832 |
+
"harvest_2026_07_04": {
|
| 833 |
+
"items": 8,
|
| 834 |
+
"aligned": 3,
|
| 835 |
+
"page_hits": 6,
|
| 836 |
+
"unique_dates": 8
|
| 837 |
+
},
|
| 838 |
+
"status": "harvested_pending_silver_pack"
|
| 839 |
+
},
|
| 840 |
+
{
|
| 841 |
+
"label": "मैथिली",
|
| 842 |
+
"aliases_seen": [
|
| 843 |
+
"मैथिली"
|
| 844 |
+
],
|
| 845 |
+
"registry_id": "maithili",
|
| 846 |
+
"primary_script": "devanagari",
|
| 847 |
+
"secondary_script": null,
|
| 848 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 849 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 850 |
+
"recognizer_ref": "devanagari",
|
| 851 |
+
"harvest_2026_07_04": {
|
| 852 |
+
"items": 8,
|
| 853 |
+
"aligned": 3,
|
| 854 |
+
"page_hits": 6,
|
| 855 |
+
"unique_dates": 8
|
| 856 |
+
},
|
| 857 |
+
"status": "harvested_pending_silver_pack"
|
| 858 |
+
},
|
| 859 |
+
{
|
| 860 |
+
"label": "संस्कृत",
|
| 861 |
+
"aliases_seen": [
|
| 862 |
+
"संस्कृत"
|
| 863 |
+
],
|
| 864 |
+
"registry_id": null,
|
| 865 |
+
"primary_script": "devanagari",
|
| 866 |
+
"secondary_script": null,
|
| 867 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 868 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 869 |
+
"recognizer_ref": "devanagari",
|
| 870 |
+
"harvest_2026_07_04": {
|
| 871 |
+
"items": 8,
|
| 872 |
+
"aligned": 3,
|
| 873 |
+
"page_hits": 6,
|
| 874 |
+
"unique_dates": 8
|
| 875 |
+
},
|
| 876 |
+
"status": "harvested_pending_silver_pack"
|
| 877 |
+
},
|
| 878 |
+
{
|
| 879 |
+
"label": "डडेल्धुरेली",
|
| 880 |
+
"aliases_seen": [
|
| 881 |
+
"डडेल्धुरेली"
|
| 882 |
+
],
|
| 883 |
+
"registry_id": null,
|
| 884 |
+
"primary_script": "devanagari",
|
| 885 |
+
"secondary_script": null,
|
| 886 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 887 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 888 |
+
"recognizer_ref": "devanagari",
|
| 889 |
+
"harvest_2026_07_04": {
|
| 890 |
+
"items": 7,
|
| 891 |
+
"aligned": 3,
|
| 892 |
+
"page_hits": 6,
|
| 893 |
+
"unique_dates": 7
|
| 894 |
+
},
|
| 895 |
+
"status": "harvested_pending_silver_pack"
|
| 896 |
+
},
|
| 897 |
+
{
|
| 898 |
+
"label": "मेचे",
|
| 899 |
+
"aliases_seen": [
|
| 900 |
+
"मेचे"
|
| 901 |
+
],
|
| 902 |
+
"registry_id": null,
|
| 903 |
+
"primary_script": "devanagari",
|
| 904 |
+
"secondary_script": null,
|
| 905 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 906 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 907 |
+
"recognizer_ref": "devanagari",
|
| 908 |
+
"harvest_2026_07_04": {
|
| 909 |
+
"items": 7,
|
| 910 |
+
"aligned": 3,
|
| 911 |
+
"page_hits": 6,
|
| 912 |
+
"unique_dates": 7
|
| 913 |
+
},
|
| 914 |
+
"status": "harvested_pending_silver_pack"
|
| 915 |
+
},
|
| 916 |
+
{
|
| 917 |
+
"label": "याक्खा",
|
| 918 |
+
"aliases_seen": [
|
| 919 |
+
"याक्ख",
|
| 920 |
+
"याक्खा"
|
| 921 |
+
],
|
| 922 |
+
"registry_id": "yakkha",
|
| 923 |
+
"primary_script": "devanagari",
|
| 924 |
+
"secondary_script": null,
|
| 925 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 926 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 927 |
+
"recognizer_ref": "devanagari",
|
| 928 |
+
"harvest_2026_07_04": {
|
| 929 |
+
"items": 7,
|
| 930 |
+
"aligned": 3,
|
| 931 |
+
"page_hits": 6,
|
| 932 |
+
"unique_dates": 7
|
| 933 |
+
},
|
| 934 |
+
"status": "harvested_pending_silver_pack"
|
| 935 |
+
},
|
| 936 |
+
{
|
| 937 |
+
"label": "शेर्पा",
|
| 938 |
+
"aliases_seen": [
|
| 939 |
+
"शेर्पा"
|
| 940 |
+
],
|
| 941 |
+
"registry_id": "sherpa",
|
| 942 |
+
"primary_script": "devanagari",
|
| 943 |
+
"secondary_script": null,
|
| 944 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 945 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 946 |
+
"recognizer_ref": "devanagari",
|
| 947 |
+
"harvest_2026_07_04": {
|
| 948 |
+
"items": 5,
|
| 949 |
+
"aligned": 3,
|
| 950 |
+
"page_hits": 5,
|
| 951 |
+
"unique_dates": 5
|
| 952 |
+
},
|
| 953 |
+
"status": "harvested_pending_silver_pack"
|
| 954 |
+
},
|
| 955 |
+
{
|
| 956 |
+
"label": "बोटे",
|
| 957 |
+
"aliases_seen": [
|
| 958 |
+
"बोटे"
|
| 959 |
+
],
|
| 960 |
+
"registry_id": null,
|
| 961 |
+
"primary_script": "devanagari",
|
| 962 |
+
"secondary_script": null,
|
| 963 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 964 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 965 |
+
"recognizer_ref": "devanagari",
|
| 966 |
+
"harvest_2026_07_04": {
|
| 967 |
+
"items": 2,
|
| 968 |
+
"aligned": 2,
|
| 969 |
+
"page_hits": 5,
|
| 970 |
+
"unique_dates": 2
|
| 971 |
+
},
|
| 972 |
+
"status": "harvested_pending_silver_pack"
|
| 973 |
+
},
|
| 974 |
+
{
|
| 975 |
+
"label": "सन्थाली",
|
| 976 |
+
"aliases_seen": [
|
| 977 |
+
"सन्थाली"
|
| 978 |
+
],
|
| 979 |
+
"registry_id": "santhali",
|
| 980 |
+
"primary_script": "ol_chiki",
|
| 981 |
+
"secondary_script": null,
|
| 982 |
+
"text_layer_class": "unclassified_pending_extraction",
|
| 983 |
+
"converter": null,
|
| 984 |
+
"recognizer_ref": "ol_chiki",
|
| 985 |
+
"harvest_2026_07_04": {
|
| 986 |
+
"items": 2,
|
| 987 |
+
"aligned": 2,
|
| 988 |
+
"page_hits": 5,
|
| 989 |
+
"unique_dates": 2
|
| 990 |
+
},
|
| 991 |
+
"status": "harvested_pending_silver_pack"
|
| 992 |
+
},
|
| 993 |
+
{
|
| 994 |
+
"label": "हायु",
|
| 995 |
+
"aliases_seen": [
|
| 996 |
+
"हायु"
|
| 997 |
+
],
|
| 998 |
+
"registry_id": null,
|
| 999 |
+
"primary_script": "devanagari",
|
| 1000 |
+
"secondary_script": null,
|
| 1001 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 1002 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 1003 |
+
"recognizer_ref": "devanagari",
|
| 1004 |
+
"harvest_2026_07_04": {
|
| 1005 |
+
"items": 2,
|
| 1006 |
+
"aligned": 2,
|
| 1007 |
+
"page_hits": 5,
|
| 1008 |
+
"unique_dates": 2
|
| 1009 |
+
},
|
| 1010 |
+
"status": "harvested_pending_silver_pack"
|
| 1011 |
+
},
|
| 1012 |
+
{
|
| 1013 |
+
"label": "उर्दू",
|
| 1014 |
+
"aliases_seen": [
|
| 1015 |
+
"उर्दू"
|
| 1016 |
+
],
|
| 1017 |
+
"registry_id": null,
|
| 1018 |
+
"primary_script": "perso_arabic",
|
| 1019 |
+
"secondary_script": null,
|
| 1020 |
+
"text_layer_class": "unclassified_pending_extraction (Perso-Arabic expected, nayanepal impossible)",
|
| 1021 |
+
"converter": null,
|
| 1022 |
+
"recognizer_ref": "perso_arabic",
|
| 1023 |
+
"harvest_2026_07_04": {
|
| 1024 |
+
"items": 8,
|
| 1025 |
+
"aligned": 2,
|
| 1026 |
+
"page_hits": 4,
|
| 1027 |
+
"unique_dates": 8
|
| 1028 |
+
},
|
| 1029 |
+
"status": "harvested_pending_silver_pack"
|
| 1030 |
+
},
|
| 1031 |
+
{
|
| 1032 |
+
"label": null,
|
| 1033 |
+
"aliases_seen": [
|
| 1034 |
+
null
|
| 1035 |
+
],
|
| 1036 |
+
"registry_id": null,
|
| 1037 |
+
"primary_script": "devanagari",
|
| 1038 |
+
"secondary_script": null,
|
| 1039 |
+
"text_layer_class": "expected_legacy_font_nayanepal (pending per-page extraction)",
|
| 1040 |
+
"converter": "nepal_ttf2utf convert_devanagari(font='nayanepal') (pending validation on this language's pages)",
|
| 1041 |
+
"recognizer_ref": "devanagari",
|
| 1042 |
+
"harvest_2026_07_04": {
|
| 1043 |
+
"items": 1,
|
| 1044 |
+
"aligned": 1,
|
| 1045 |
+
"page_hits": 2,
|
| 1046 |
+
"unique_dates": 1
|
| 1047 |
+
},
|
| 1048 |
+
"status": "harvested_pending_silver_pack"
|
| 1049 |
+
}
|
| 1050 |
+
]
|
| 1051 |
+
}
|
ocr-tech-mps-integration/configs/nepali-table-name-lexicon-v1.txt
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Small Nepali table-name lexicon for guarded OCR correction.
|
| 2 |
+
# Use only with provenance in the model card; this is not a general language model.
|
| 3 |
+
किरण
|
| 4 |
+
माया
|
| 5 |
+
राम
|
| 6 |
+
सीता
|
| 7 |
+
हरी
|
| 8 |
+
गीता
|
| 9 |
+
सोम
|
| 10 |
+
रीता
|
ocr-tech-mps-integration/configs/recognizer-eval-pack-normalization-v1.json
ADDED
|
@@ -0,0 +1,165 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"version": "recognizer-eval-pack-normalization-v1",
|
| 3 |
+
"updated": "2026-06-19",
|
| 4 |
+
"packs": [
|
| 5 |
+
{
|
| 6 |
+
"id": "newa-zenodo-6967421-v1",
|
| 7 |
+
"label_file": "data/newa-real-eval-pack/eval.txt",
|
| 8 |
+
"data_dir": "data/newa-real-eval-pack",
|
| 9 |
+
"manifest_file": "data/newa-real-eval-pack/manifest.jsonl",
|
| 10 |
+
"summary_file": "data/newa-real-eval-pack/summary.json",
|
| 11 |
+
"out": "outputs/eval-packs/newa-zenodo-6967421-v1",
|
| 12 |
+
"dataset": "newa-zenodo-6967421-prachalit-real",
|
| 13 |
+
"language": "newa",
|
| 14 |
+
"script": "Newa",
|
| 15 |
+
"domain": "real_handwritten",
|
| 16 |
+
"gt_status": "machine_converted",
|
| 17 |
+
"split": "eval",
|
| 18 |
+
"eval_pack_version": "newa-zenodo-6967421-v1",
|
| 19 |
+
"frozen_eval_pack": true,
|
| 20 |
+
"unseen_for_selection": false,
|
| 21 |
+
"claim_eligible": false,
|
| 22 |
+
"claim_blockers": [
|
| 23 |
+
"eval pack is not declared unseen for model/checkpoint selection",
|
| 24 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 25 |
+
"row-level claim eligibility is false"
|
| 26 |
+
]
|
| 27 |
+
},
|
| 28 |
+
{
|
| 29 |
+
"id": "sunuwar-herald-real-v1",
|
| 30 |
+
"label_file": "data/sunuwar-real-eval-pack/real_eval.txt",
|
| 31 |
+
"data_dir": "data/sunuwar-real-eval-pack",
|
| 32 |
+
"manifest_file": "data/sunuwar-real-eval-pack/manifests/recognizer-eval.jsonl",
|
| 33 |
+
"summary_file": "data/sunuwar-real-eval-pack/README.json",
|
| 34 |
+
"out": "outputs/eval-packs/sunuwar-herald-real-v1",
|
| 35 |
+
"dataset": "sikkim-herald-sunuwar-real",
|
| 36 |
+
"language": "sunuwar",
|
| 37 |
+
"script": "Sunuwar",
|
| 38 |
+
"domain": "real_print",
|
| 39 |
+
"gt_status": "machine_converted",
|
| 40 |
+
"split": "eval",
|
| 41 |
+
"eval_pack_version": "sunuwar-herald-real-v1",
|
| 42 |
+
"frozen_eval_pack": true,
|
| 43 |
+
"unseen_for_selection": true,
|
| 44 |
+
"claim_eligible": false,
|
| 45 |
+
"claim_blockers": [
|
| 46 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 47 |
+
"row-level claim eligibility is false"
|
| 48 |
+
]
|
| 49 |
+
},
|
| 50 |
+
{
|
| 51 |
+
"id": "lepcha-herald-real-v1",
|
| 52 |
+
"label_file": "data/lepcha-real-eval-pack/real_eval.txt",
|
| 53 |
+
"data_dir": "data/lepcha-real-eval-pack",
|
| 54 |
+
"manifest_file": "data/lepcha-real-eval-pack/manifests/recognizer-eval.jsonl",
|
| 55 |
+
"summary_file": "data/lepcha-real-eval-pack/README.json",
|
| 56 |
+
"out": "outputs/eval-packs/lepcha-herald-real-v1",
|
| 57 |
+
"dataset": "sikkim-herald-lepcha-real",
|
| 58 |
+
"language": "lepcha",
|
| 59 |
+
"script": "Lepcha",
|
| 60 |
+
"domain": "real_print",
|
| 61 |
+
"gt_status": "machine_converted",
|
| 62 |
+
"split": "eval",
|
| 63 |
+
"eval_pack_version": "lepcha-herald-real-v1",
|
| 64 |
+
"frozen_eval_pack": true,
|
| 65 |
+
"unseen_for_selection": true,
|
| 66 |
+
"claim_eligible": false,
|
| 67 |
+
"claim_blockers": [
|
| 68 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 69 |
+
"row-level claim eligibility is false"
|
| 70 |
+
]
|
| 71 |
+
},
|
| 72 |
+
{
|
| 73 |
+
"id": "limbu-gorkhapatra-real-v1",
|
| 74 |
+
"label_file": "data/limbu-real-crops/labels.txt",
|
| 75 |
+
"data_dir": "data/limbu-real-crops",
|
| 76 |
+
"out": "outputs/eval-packs/limbu-gorkhapatra-real-v1",
|
| 77 |
+
"dataset": "gorkhapatra-limbu-real-crops",
|
| 78 |
+
"language": "limbu",
|
| 79 |
+
"script": "Limbu",
|
| 80 |
+
"domain": "real_print",
|
| 81 |
+
"gt_status": "machine_converted",
|
| 82 |
+
"split": "eval",
|
| 83 |
+
"eval_pack_version": "limbu-gorkhapatra-real-v1",
|
| 84 |
+
"frozen_eval_pack": true,
|
| 85 |
+
"unseen_for_selection": false,
|
| 86 |
+
"claim_eligible": false,
|
| 87 |
+
"claim_blockers": [
|
| 88 |
+
"source pack lacks canonical manifest and summary",
|
| 89 |
+
"eval pack is not declared unseen for model/checkpoint selection",
|
| 90 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 91 |
+
"row-level claim eligibility is false"
|
| 92 |
+
]
|
| 93 |
+
},
|
| 94 |
+
{
|
| 95 |
+
"id": "limbu-devanagari-cdc-line-crop-corpus-draft-v1",
|
| 96 |
+
"label_file": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/eval.txt",
|
| 97 |
+
"data_dir": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21",
|
| 98 |
+
"manifest_file": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/manifests/recognizer-eval.jsonl",
|
| 99 |
+
"summary_file": "outputs/limbu-recognizer-eval-packs/limbu-devanagari-cdc-line-crop-corpus-devclean-2026-06-21/summaries/eval.summary.json",
|
| 100 |
+
"out": "outputs/eval-packs/limbu-devanagari-cdc-line-crop-corpus-draft-v1",
|
| 101 |
+
"dataset": "limbu-devanagari-cdc-line-crop-corpus-draft",
|
| 102 |
+
"language": "limbu",
|
| 103 |
+
"script": "Devanagari",
|
| 104 |
+
"domain": "real_print",
|
| 105 |
+
"gt_status": "curriculum_text_unreviewed",
|
| 106 |
+
"split": "eval",
|
| 107 |
+
"eval_pack_version": "limbu-devanagari-cdc-line-crop-corpus-draft-v1",
|
| 108 |
+
"frozen_eval_pack": false,
|
| 109 |
+
"unseen_for_selection": true,
|
| 110 |
+
"claim_eligible": false,
|
| 111 |
+
"claim_blockers": [
|
| 112 |
+
"eval pack is not frozen",
|
| 113 |
+
"GT is unreviewed curriculum text and lacks human/native-reader calibration",
|
| 114 |
+
"row-level claim eligibility is false",
|
| 115 |
+
"trained Devanagari recognizer checkpoint has not completed held-out admission"
|
| 116 |
+
]
|
| 117 |
+
},
|
| 118 |
+
{
|
| 119 |
+
"id": "kiratrai-herald-real-v1",
|
| 120 |
+
"label_file": "data/kiratrai-real-eval-pack/eval.txt",
|
| 121 |
+
"data_dir": "data/kiratrai-real-eval-pack",
|
| 122 |
+
"summary_file": "data/kiratrai-real-eval-pack/summary.json",
|
| 123 |
+
"out": "outputs/eval-packs/kiratrai-herald-real-v1",
|
| 124 |
+
"dataset": "sikkim-herald-kiratrai-real",
|
| 125 |
+
"language": "kiratrai",
|
| 126 |
+
"script": "KiratRai",
|
| 127 |
+
"domain": "real_print",
|
| 128 |
+
"gt_status": "machine_converted",
|
| 129 |
+
"split": "eval",
|
| 130 |
+
"eval_pack_version": "kiratrai-herald-real-v1",
|
| 131 |
+
"frozen_eval_pack": true,
|
| 132 |
+
"unseen_for_selection": true,
|
| 133 |
+
"claim_eligible": false,
|
| 134 |
+
"claim_blockers": [
|
| 135 |
+
"source pack lacks canonical manifest",
|
| 136 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 137 |
+
"row-level claim eligibility is false"
|
| 138 |
+
]
|
| 139 |
+
},
|
| 140 |
+
{
|
| 141 |
+
"id": "gorkhapatra-devanagari-real-v1",
|
| 142 |
+
"label_file": "data/gorkhapatra-devanagari-real/labels.txt",
|
| 143 |
+
"data_dir": "data/gorkhapatra-devanagari-real",
|
| 144 |
+
"summary_file": "data/gorkhapatra-devanagari-real/summary.json",
|
| 145 |
+
"out": "outputs/eval-packs/gorkhapatra-devanagari-real-v1",
|
| 146 |
+
"dataset": "gorkhapatra-devanagari-real",
|
| 147 |
+
"language": "devanagari_shared",
|
| 148 |
+
"script": "Devanagari",
|
| 149 |
+
"domain": "real_print",
|
| 150 |
+
"gt_status": "machine_converted",
|
| 151 |
+
"split": "eval",
|
| 152 |
+
"eval_pack_version": "gorkhapatra-devanagari-real-v1",
|
| 153 |
+
"frozen_eval_pack": true,
|
| 154 |
+
"unseen_for_selection": false,
|
| 155 |
+
"claim_eligible": false,
|
| 156 |
+
"claim_blockers": [
|
| 157 |
+
"source pack lacks canonical manifest",
|
| 158 |
+
"eval pack is not declared unseen for model/checkpoint selection",
|
| 159 |
+
"GT is machine-converted and lacks native-reader calibration",
|
| 160 |
+
"row-level claim eligibility is false",
|
| 161 |
+
"shared Devanagari pack still needs per-language source slices"
|
| 162 |
+
]
|
| 163 |
+
}
|
| 164 |
+
]
|
| 165 |
+
}
|
ocr-tech-mps-integration/configs/validation-complex-table-surya.json
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"candidate": "candidate",
|
| 3 |
+
"min_samples": 40,
|
| 4 |
+
"required_slices": {
|
| 5 |
+
"complex_table": 40,
|
| 6 |
+
"table": 40,
|
| 7 |
+
"nepali": 40,
|
| 8 |
+
"english": 40
|
| 9 |
+
},
|
| 10 |
+
"bootstrap_iterations": 1000,
|
| 11 |
+
"bootstrap_confidence": 0.95,
|
| 12 |
+
"random_seed": 13,
|
| 13 |
+
"gates": [
|
| 14 |
+
{
|
| 15 |
+
"baseline": "surya",
|
| 16 |
+
"metric": "table_cell_f1",
|
| 17 |
+
"direction": "higher",
|
| 18 |
+
"slices": ["complex_table", "table"],
|
| 19 |
+
"min_pairs": 40,
|
| 20 |
+
"min_absolute_margin": 0.005
|
| 21 |
+
},
|
| 22 |
+
{
|
| 23 |
+
"baseline": "surya",
|
| 24 |
+
"metric": "table_exact_match",
|
| 25 |
+
"direction": "higher",
|
| 26 |
+
"slices": ["complex_table", "table"],
|
| 27 |
+
"min_pairs": 40,
|
| 28 |
+
"min_absolute_margin": 0.0,
|
| 29 |
+
"require_ci_above_zero": false
|
| 30 |
+
},
|
| 31 |
+
{
|
| 32 |
+
"baseline": "surya",
|
| 33 |
+
"metric": "table_structure_similarity",
|
| 34 |
+
"direction": "higher",
|
| 35 |
+
"slices": ["complex_table", "table"],
|
| 36 |
+
"min_pairs": 40,
|
| 37 |
+
"min_absolute_margin": 0.0,
|
| 38 |
+
"require_ci_above_zero": false
|
| 39 |
+
},
|
| 40 |
+
{
|
| 41 |
+
"baseline": "surya",
|
| 42 |
+
"metric": "table_teds_structure",
|
| 43 |
+
"direction": "higher",
|
| 44 |
+
"slices": ["complex_table", "table"],
|
| 45 |
+
"min_pairs": 40,
|
| 46 |
+
"min_absolute_margin": 0.0
|
| 47 |
+
},
|
| 48 |
+
{
|
| 49 |
+
"baseline": "surya",
|
| 50 |
+
"metric": "table_teds_text",
|
| 51 |
+
"direction": "higher",
|
| 52 |
+
"slices": ["complex_table", "table"],
|
| 53 |
+
"min_pairs": 40,
|
| 54 |
+
"min_absolute_margin": 0.005
|
| 55 |
+
}
|
| 56 |
+
]
|
| 57 |
+
}
|
ocr-tech-mps-integration/configs/validation-hard15-surya.json
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"candidate": "candidate",
|
| 3 |
+
"min_samples": 15,
|
| 4 |
+
"required_slices": {
|
| 5 |
+
"hard_eval": 15,
|
| 6 |
+
"table": 8,
|
| 7 |
+
"figure": 7,
|
| 8 |
+
"nepali": 15,
|
| 9 |
+
"english": 15
|
| 10 |
+
},
|
| 11 |
+
"bootstrap_iterations": 1000,
|
| 12 |
+
"bootstrap_confidence": 0.95,
|
| 13 |
+
"random_seed": 13,
|
| 14 |
+
"gates": [
|
| 15 |
+
{
|
| 16 |
+
"baseline": "surya",
|
| 17 |
+
"metric": "cer",
|
| 18 |
+
"direction": "lower",
|
| 19 |
+
"slices": ["hard_eval"],
|
| 20 |
+
"min_pairs": 15,
|
| 21 |
+
"min_absolute_margin": 0.02,
|
| 22 |
+
"max_p_value": 0.05
|
| 23 |
+
},
|
| 24 |
+
{
|
| 25 |
+
"baseline": "surya",
|
| 26 |
+
"metric": "cer",
|
| 27 |
+
"direction": "lower",
|
| 28 |
+
"slices": ["hard_eval", "table"],
|
| 29 |
+
"min_pairs": 8,
|
| 30 |
+
"min_absolute_margin": 0.02,
|
| 31 |
+
"max_p_value": 0.05
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"baseline": "surya",
|
| 35 |
+
"metric": "cer",
|
| 36 |
+
"direction": "lower",
|
| 37 |
+
"slices": ["hard_eval", "figure"],
|
| 38 |
+
"min_pairs": 7,
|
| 39 |
+
"min_absolute_margin": 0.02,
|
| 40 |
+
"max_p_value": 0.05
|
| 41 |
+
},
|
| 42 |
+
{
|
| 43 |
+
"baseline": "surya",
|
| 44 |
+
"metric": "table_cell_f1",
|
| 45 |
+
"direction": "higher",
|
| 46 |
+
"slices": ["hard_eval", "table"],
|
| 47 |
+
"min_pairs": 8,
|
| 48 |
+
"min_absolute_margin": 0.01
|
| 49 |
+
},
|
| 50 |
+
{
|
| 51 |
+
"baseline": "surya",
|
| 52 |
+
"metric": "figure_caption_cer",
|
| 53 |
+
"direction": "lower",
|
| 54 |
+
"slices": ["hard_eval", "figure"],
|
| 55 |
+
"min_pairs": 7,
|
| 56 |
+
"min_absolute_margin": 0.01
|
| 57 |
+
}
|
| 58 |
+
]
|
| 59 |
+
}
|
ocr-tech-mps-integration/configs/validation-hf-recognizer-next-v1-admission.json
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"candidate": "candidate",
|
| 3 |
+
"min_samples": 244,
|
| 4 |
+
"required_slices": {
|
| 5 |
+
"english": 214,
|
| 6 |
+
"line_crop": 150,
|
| 7 |
+
"nepali": 94,
|
| 8 |
+
"real": 15,
|
| 9 |
+
"synthetic": 165
|
| 10 |
+
},
|
| 11 |
+
"bootstrap_iterations": 1000,
|
| 12 |
+
"bootstrap_confidence": 0.95,
|
| 13 |
+
"random_seed": 13,
|
| 14 |
+
"gates": [
|
| 15 |
+
{
|
| 16 |
+
"baseline": "tesseract",
|
| 17 |
+
"metric": "cer",
|
| 18 |
+
"direction": "lower",
|
| 19 |
+
"min_pairs": 180,
|
| 20 |
+
"min_absolute_margin": 0.05,
|
| 21 |
+
"require_ci_above_zero": false
|
| 22 |
+
},
|
| 23 |
+
{
|
| 24 |
+
"baseline": "tesseract",
|
| 25 |
+
"metric": "cer",
|
| 26 |
+
"direction": "lower",
|
| 27 |
+
"slices": ["nepali"],
|
| 28 |
+
"min_pairs": 30,
|
| 29 |
+
"min_absolute_margin": 0.05,
|
| 30 |
+
"require_ci_above_zero": false
|
| 31 |
+
},
|
| 32 |
+
{
|
| 33 |
+
"baseline": "stock-paddle",
|
| 34 |
+
"metric": "cer",
|
| 35 |
+
"direction": "lower",
|
| 36 |
+
"min_pairs": 180,
|
| 37 |
+
"min_absolute_margin": 0.10,
|
| 38 |
+
"min_pair_win_rate": 0.80,
|
| 39 |
+
"require_ci_above_zero": false
|
| 40 |
+
},
|
| 41 |
+
{
|
| 42 |
+
"baseline": "stock-paddle",
|
| 43 |
+
"metric": "cer",
|
| 44 |
+
"direction": "lower",
|
| 45 |
+
"slices": ["english", "line_crop"],
|
| 46 |
+
"min_pairs": 150,
|
| 47 |
+
"min_absolute_margin": 0.10,
|
| 48 |
+
"min_pair_win_rate": 0.80,
|
| 49 |
+
"require_ci_above_zero": false
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"baseline": "stock-paddle",
|
| 53 |
+
"metric": "cer",
|
| 54 |
+
"direction": "lower",
|
| 55 |
+
"slices": ["nepali"],
|
| 56 |
+
"min_pairs": 30,
|
| 57 |
+
"min_absolute_margin": 0.10,
|
| 58 |
+
"min_pair_win_rate": 0.75,
|
| 59 |
+
"require_ci_above_zero": false
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"baseline": "surya",
|
| 63 |
+
"metric": "cer",
|
| 64 |
+
"direction": "lower",
|
| 65 |
+
"min_pairs": 179,
|
| 66 |
+
"min_absolute_margin": 0.10,
|
| 67 |
+
"require_ci_above_zero": false
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"baseline": "surya",
|
| 71 |
+
"metric": "cer",
|
| 72 |
+
"direction": "lower",
|
| 73 |
+
"slices": ["nepali"],
|
| 74 |
+
"min_pairs": 30,
|
| 75 |
+
"min_absolute_margin": 0.05,
|
| 76 |
+
"require_ci_above_zero": false
|
| 77 |
+
}
|
| 78 |
+
]
|
| 79 |
+
}
|
ocr-tech-mps-integration/configs/validation-internal-sota-v1.json
ADDED
|
@@ -0,0 +1,120 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"candidate": "candidate",
|
| 3 |
+
"min_samples": 100,
|
| 4 |
+
"required_slices": {
|
| 5 |
+
"nepali": 50,
|
| 6 |
+
"english": 50,
|
| 7 |
+
"table": 20,
|
| 8 |
+
"reading_order": 20,
|
| 9 |
+
"figure": 20
|
| 10 |
+
},
|
| 11 |
+
"bootstrap_iterations": 1000,
|
| 12 |
+
"bootstrap_confidence": 0.95,
|
| 13 |
+
"random_seed": 13,
|
| 14 |
+
"require_full_eval_manifest_coverage": true,
|
| 15 |
+
"gates": [
|
| 16 |
+
{
|
| 17 |
+
"baseline": "tesseract",
|
| 18 |
+
"metric": "cer",
|
| 19 |
+
"direction": "lower",
|
| 20 |
+
"slices": ["nepali"],
|
| 21 |
+
"min_pairs": 50,
|
| 22 |
+
"min_absolute_margin": 0.02,
|
| 23 |
+
"min_pair_win_rate": 0.55,
|
| 24 |
+
"max_p_value": 0.05
|
| 25 |
+
},
|
| 26 |
+
{
|
| 27 |
+
"baseline": "tesseract",
|
| 28 |
+
"metric": "wer",
|
| 29 |
+
"direction": "lower",
|
| 30 |
+
"slices": ["nepali"],
|
| 31 |
+
"min_pairs": 50,
|
| 32 |
+
"min_absolute_margin": 0.01,
|
| 33 |
+
"min_pair_win_rate": 0.55,
|
| 34 |
+
"max_p_value": 0.05
|
| 35 |
+
},
|
| 36 |
+
{
|
| 37 |
+
"baseline": "stock-paddle",
|
| 38 |
+
"metric": "cer",
|
| 39 |
+
"direction": "lower",
|
| 40 |
+
"slices": ["nepali"],
|
| 41 |
+
"min_pairs": 50,
|
| 42 |
+
"min_absolute_margin": 0.01,
|
| 43 |
+
"min_pair_win_rate": 0.55,
|
| 44 |
+
"max_p_value": 0.05
|
| 45 |
+
},
|
| 46 |
+
{
|
| 47 |
+
"baseline": "stock-paddle",
|
| 48 |
+
"metric": "cer",
|
| 49 |
+
"direction": "lower",
|
| 50 |
+
"slices": ["english"],
|
| 51 |
+
"min_pairs": 50,
|
| 52 |
+
"min_absolute_margin": 0.0,
|
| 53 |
+
"require_ci_above_zero": false
|
| 54 |
+
},
|
| 55 |
+
{
|
| 56 |
+
"baseline": "stock-paddle",
|
| 57 |
+
"metric": "wer",
|
| 58 |
+
"direction": "lower",
|
| 59 |
+
"slices": ["english"],
|
| 60 |
+
"min_pairs": 50,
|
| 61 |
+
"min_absolute_margin": 0.0,
|
| 62 |
+
"require_ci_above_zero": false
|
| 63 |
+
},
|
| 64 |
+
{
|
| 65 |
+
"baseline": "glm-ocr",
|
| 66 |
+
"metric": "cer",
|
| 67 |
+
"direction": "lower",
|
| 68 |
+
"slices": ["nepali"],
|
| 69 |
+
"min_pairs": 50,
|
| 70 |
+
"min_absolute_margin": 0.01,
|
| 71 |
+
"min_pair_win_rate": 0.55,
|
| 72 |
+
"max_p_value": 0.05
|
| 73 |
+
},
|
| 74 |
+
{
|
| 75 |
+
"baseline": "surya",
|
| 76 |
+
"metric": "table_cell_f1",
|
| 77 |
+
"direction": "higher",
|
| 78 |
+
"slices": ["table"],
|
| 79 |
+
"min_pairs": 20,
|
| 80 |
+
"min_absolute_margin": 0.01,
|
| 81 |
+
"min_pair_win_rate": 0.55
|
| 82 |
+
},
|
| 83 |
+
{
|
| 84 |
+
"baseline": "paddleocr-vl",
|
| 85 |
+
"metric": "reading_order_pair_accuracy",
|
| 86 |
+
"direction": "higher",
|
| 87 |
+
"slices": ["reading_order"],
|
| 88 |
+
"min_pairs": 20,
|
| 89 |
+
"min_absolute_margin": 0.01,
|
| 90 |
+
"min_pair_win_rate": 0.55
|
| 91 |
+
},
|
| 92 |
+
{
|
| 93 |
+
"baseline": "paddleocr-vl",
|
| 94 |
+
"metric": "document_reconstruction_similarity",
|
| 95 |
+
"direction": "higher",
|
| 96 |
+
"slices": ["figure"],
|
| 97 |
+
"min_pairs": 20,
|
| 98 |
+
"min_absolute_margin": 0.01,
|
| 99 |
+
"min_pair_win_rate": 0.55
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"baseline": "surya",
|
| 103 |
+
"metric": "figure_detection_f1",
|
| 104 |
+
"direction": "higher",
|
| 105 |
+
"slices": ["figure"],
|
| 106 |
+
"min_pairs": 20,
|
| 107 |
+
"min_absolute_margin": 0.0,
|
| 108 |
+
"require_ci_above_zero": false
|
| 109 |
+
},
|
| 110 |
+
{
|
| 111 |
+
"baseline": "surya",
|
| 112 |
+
"metric": "figure_caption_cer",
|
| 113 |
+
"direction": "lower",
|
| 114 |
+
"slices": ["figure"],
|
| 115 |
+
"min_pairs": 20,
|
| 116 |
+
"min_absolute_margin": 0.01,
|
| 117 |
+
"min_pair_win_rate": 0.55
|
| 118 |
+
}
|
| 119 |
+
]
|
| 120 |
+
}
|
ocr-tech-mps-integration/configs/validation-paddle-hard50-tesseract.json
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"candidate": "candidate",
|
| 3 |
+
"min_samples": 50,
|
| 4 |
+
"required_slices": {
|
| 5 |
+
"paddle_hard_eval": 50,
|
| 6 |
+
"synthetic": 50,
|
| 7 |
+
"english": 38,
|
| 8 |
+
"line_crop": 38,
|
| 9 |
+
"nepali": 12
|
| 10 |
+
},
|
| 11 |
+
"bootstrap_iterations": 1000,
|
| 12 |
+
"bootstrap_confidence": 0.95,
|
| 13 |
+
"random_seed": 13,
|
| 14 |
+
"gates": [
|
| 15 |
+
{
|
| 16 |
+
"baseline": "tesseract",
|
| 17 |
+
"metric": "cer",
|
| 18 |
+
"direction": "lower",
|
| 19 |
+
"slices": ["paddle_hard_eval"],
|
| 20 |
+
"min_pairs": 50,
|
| 21 |
+
"min_absolute_margin": 0.0,
|
| 22 |
+
"require_ci_above_zero": false
|
| 23 |
+
},
|
| 24 |
+
{
|
| 25 |
+
"baseline": "tesseract",
|
| 26 |
+
"metric": "cer",
|
| 27 |
+
"direction": "lower",
|
| 28 |
+
"slices": ["paddle_hard_eval", "english", "line_crop"],
|
| 29 |
+
"min_pairs": 38,
|
| 30 |
+
"min_absolute_margin": 0.0,
|
| 31 |
+
"require_ci_above_zero": false
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"baseline": "tesseract",
|
| 35 |
+
"metric": "wer",
|
| 36 |
+
"direction": "lower",
|
| 37 |
+
"slices": ["paddle_hard_eval", "english", "line_crop"],
|
| 38 |
+
"min_pairs": 38,
|
| 39 |
+
"min_absolute_margin": 0.0,
|
| 40 |
+
"require_ci_above_zero": false
|
| 41 |
+
},
|
| 42 |
+
{
|
| 43 |
+
"baseline": "tesseract",
|
| 44 |
+
"metric": "wer",
|
| 45 |
+
"direction": "lower",
|
| 46 |
+
"slices": ["paddle_hard_eval", "nepali"],
|
| 47 |
+
"min_pairs": 12,
|
| 48 |
+
"min_absolute_margin": 0.05,
|
| 49 |
+
"require_ci_above_zero": false
|
| 50 |
+
}
|
| 51 |
+
]
|
| 52 |
+
}
|
ocr-tech-mps-integration/configs/validation-public-omnidocbench-sota-v1.json
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"candidate": "candidate",
|
| 3 |
+
"min_samples": 100,
|
| 4 |
+
"required_slices": {
|
| 5 |
+
"public": 100,
|
| 6 |
+
"document_parsing": 100,
|
| 7 |
+
"table": 40
|
| 8 |
+
},
|
| 9 |
+
"bootstrap_iterations": 1000,
|
| 10 |
+
"bootstrap_confidence": 0.95,
|
| 11 |
+
"random_seed": 13,
|
| 12 |
+
"require_full_eval_manifest_coverage": true,
|
| 13 |
+
"gates": [
|
| 14 |
+
{
|
| 15 |
+
"baseline": "surya",
|
| 16 |
+
"metric": "cer",
|
| 17 |
+
"direction": "lower",
|
| 18 |
+
"slices": ["public", "document_parsing"],
|
| 19 |
+
"min_pairs": 100,
|
| 20 |
+
"min_absolute_margin": 0.005,
|
| 21 |
+
"min_pair_win_rate": 0.55
|
| 22 |
+
},
|
| 23 |
+
{
|
| 24 |
+
"baseline": "surya",
|
| 25 |
+
"metric": "table_cell_f1",
|
| 26 |
+
"direction": "higher",
|
| 27 |
+
"slices": ["public", "table"],
|
| 28 |
+
"min_pairs": 40,
|
| 29 |
+
"min_absolute_margin": 0.005,
|
| 30 |
+
"min_pair_win_rate": 0.55
|
| 31 |
+
},
|
| 32 |
+
{
|
| 33 |
+
"baseline": "surya",
|
| 34 |
+
"metric": "table_structure_similarity",
|
| 35 |
+
"direction": "higher",
|
| 36 |
+
"slices": ["public", "document_parsing"],
|
| 37 |
+
"min_pairs": 40,
|
| 38 |
+
"min_absolute_margin": 0.0,
|
| 39 |
+
"require_ci_above_zero": false
|
| 40 |
+
},
|
| 41 |
+
{
|
| 42 |
+
"baseline": "paddleocr-vl",
|
| 43 |
+
"metric": "document_reconstruction_similarity",
|
| 44 |
+
"direction": "higher",
|
| 45 |
+
"slices": ["public", "document_parsing"],
|
| 46 |
+
"min_pairs": 100,
|
| 47 |
+
"min_absolute_margin": 0.005,
|
| 48 |
+
"min_pair_win_rate": 0.55
|
| 49 |
+
},
|
| 50 |
+
{
|
| 51 |
+
"baseline": "paddleocr-vl",
|
| 52 |
+
"metric": "reading_order_pair_accuracy",
|
| 53 |
+
"direction": "higher",
|
| 54 |
+
"slices": ["public", "document_parsing"],
|
| 55 |
+
"min_pairs": 100,
|
| 56 |
+
"min_absolute_margin": 0.005,
|
| 57 |
+
"min_pair_win_rate": 0.55
|
| 58 |
+
}
|
| 59 |
+
]
|
| 60 |
+
}
|
ocr-tech-mps-integration/configs/validation-public-omnidocbench-surya.json
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"candidate": "candidate",
|
| 3 |
+
"min_samples": 100,
|
| 4 |
+
"required_slices": {
|
| 5 |
+
"public": 100,
|
| 6 |
+
"document_parsing": 100,
|
| 7 |
+
"table": 40
|
| 8 |
+
},
|
| 9 |
+
"bootstrap_iterations": 1000,
|
| 10 |
+
"bootstrap_confidence": 0.95,
|
| 11 |
+
"random_seed": 13,
|
| 12 |
+
"gates": [
|
| 13 |
+
{
|
| 14 |
+
"baseline": "surya",
|
| 15 |
+
"metric": "cer",
|
| 16 |
+
"direction": "lower",
|
| 17 |
+
"slices": ["public", "document_parsing"],
|
| 18 |
+
"min_pairs": 100,
|
| 19 |
+
"min_absolute_margin": 0.005
|
| 20 |
+
},
|
| 21 |
+
{
|
| 22 |
+
"baseline": "surya",
|
| 23 |
+
"metric": "table_cell_f1",
|
| 24 |
+
"direction": "higher",
|
| 25 |
+
"slices": ["public", "table"],
|
| 26 |
+
"min_pairs": 40,
|
| 27 |
+
"min_absolute_margin": 0.005
|
| 28 |
+
},
|
| 29 |
+
{
|
| 30 |
+
"baseline": "surya",
|
| 31 |
+
"metric": "table_teds_structure",
|
| 32 |
+
"direction": "higher",
|
| 33 |
+
"slices": ["public", "table"],
|
| 34 |
+
"min_pairs": 40,
|
| 35 |
+
"min_absolute_margin": 0.005
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"baseline": "surya",
|
| 39 |
+
"metric": "table_teds_text",
|
| 40 |
+
"direction": "higher",
|
| 41 |
+
"slices": ["public", "table"],
|
| 42 |
+
"min_pairs": 40,
|
| 43 |
+
"min_absolute_margin": 0.005
|
| 44 |
+
}
|
| 45 |
+
]
|
| 46 |
+
}
|
ocr-tech-mps-integration/configs/validation-public-real5-sota-v1.json
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"candidate": "candidate",
|
| 3 |
+
"min_samples": 100,
|
| 4 |
+
"required_slices": {
|
| 5 |
+
"public": 100
|
| 6 |
+
},
|
| 7 |
+
"bootstrap_iterations": 1000,
|
| 8 |
+
"bootstrap_confidence": 0.95,
|
| 9 |
+
"random_seed": 13,
|
| 10 |
+
"require_full_eval_manifest_coverage": true,
|
| 11 |
+
"gates": [
|
| 12 |
+
{
|
| 13 |
+
"baseline": "surya",
|
| 14 |
+
"metric": "cer",
|
| 15 |
+
"direction": "lower",
|
| 16 |
+
"slices": ["public"],
|
| 17 |
+
"min_pairs": 100,
|
| 18 |
+
"min_absolute_margin": 0.005,
|
| 19 |
+
"min_pair_win_rate": 0.55
|
| 20 |
+
},
|
| 21 |
+
{
|
| 22 |
+
"baseline": "surya",
|
| 23 |
+
"metric": "wer",
|
| 24 |
+
"direction": "lower",
|
| 25 |
+
"slices": ["public"],
|
| 26 |
+
"min_pairs": 100,
|
| 27 |
+
"min_absolute_margin": 0.005,
|
| 28 |
+
"min_pair_win_rate": 0.55
|
| 29 |
+
},
|
| 30 |
+
{
|
| 31 |
+
"baseline": "glm-ocr",
|
| 32 |
+
"metric": "cer",
|
| 33 |
+
"direction": "lower",
|
| 34 |
+
"slices": ["public"],
|
| 35 |
+
"min_pairs": 100,
|
| 36 |
+
"min_absolute_margin": 0.005,
|
| 37 |
+
"min_pair_win_rate": 0.55
|
| 38 |
+
},
|
| 39 |
+
{
|
| 40 |
+
"baseline": "glm-ocr",
|
| 41 |
+
"metric": "wer",
|
| 42 |
+
"direction": "lower",
|
| 43 |
+
"slices": ["public"],
|
| 44 |
+
"min_pairs": 100,
|
| 45 |
+
"min_absolute_margin": 0.005,
|
| 46 |
+
"min_pair_win_rate": 0.55
|
| 47 |
+
},
|
| 48 |
+
{
|
| 49 |
+
"baseline": "paddleocr-vl",
|
| 50 |
+
"metric": "document_reconstruction_similarity",
|
| 51 |
+
"direction": "higher",
|
| 52 |
+
"slices": ["public"],
|
| 53 |
+
"min_pairs": 100,
|
| 54 |
+
"min_absolute_margin": 0.005,
|
| 55 |
+
"min_pair_win_rate": 0.55
|
| 56 |
+
},
|
| 57 |
+
{
|
| 58 |
+
"baseline": "paddleocr-vl",
|
| 59 |
+
"metric": "reading_order_pair_accuracy",
|
| 60 |
+
"direction": "higher",
|
| 61 |
+
"slices": ["public"],
|
| 62 |
+
"min_pairs": 100,
|
| 63 |
+
"min_absolute_margin": 0.005,
|
| 64 |
+
"min_pair_win_rate": 0.55
|
| 65 |
+
}
|
| 66 |
+
]
|
| 67 |
+
}
|
ocr-tech-mps-integration/configs/validation-table-ledger-surya.json
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"candidate": "candidate",
|
| 3 |
+
"min_samples": 40,
|
| 4 |
+
"required_slices": {
|
| 5 |
+
"table": 40,
|
| 6 |
+
"nepali": 40,
|
| 7 |
+
"english": 40
|
| 8 |
+
},
|
| 9 |
+
"bootstrap_iterations": 1000,
|
| 10 |
+
"bootstrap_confidence": 0.95,
|
| 11 |
+
"random_seed": 13,
|
| 12 |
+
"gates": [
|
| 13 |
+
{
|
| 14 |
+
"baseline": "surya",
|
| 15 |
+
"metric": "table_cell_f1",
|
| 16 |
+
"direction": "higher",
|
| 17 |
+
"slices": ["table"],
|
| 18 |
+
"min_pairs": 40,
|
| 19 |
+
"min_absolute_margin": 0.01
|
| 20 |
+
},
|
| 21 |
+
{
|
| 22 |
+
"baseline": "surya",
|
| 23 |
+
"metric": "table_exact_match",
|
| 24 |
+
"direction": "higher",
|
| 25 |
+
"slices": ["table"],
|
| 26 |
+
"min_pairs": 40,
|
| 27 |
+
"min_absolute_margin": 0.01
|
| 28 |
+
},
|
| 29 |
+
{
|
| 30 |
+
"baseline": "surya",
|
| 31 |
+
"metric": "table_structure_similarity",
|
| 32 |
+
"direction": "higher",
|
| 33 |
+
"slices": ["table"],
|
| 34 |
+
"min_pairs": 40,
|
| 35 |
+
"min_absolute_margin": 0.0,
|
| 36 |
+
"require_ci_above_zero": false
|
| 37 |
+
}
|
| 38 |
+
]
|
| 39 |
+
}
|
ocr-tech-mps-integration/data/dictionaries/devanagari-v1.txt
ADDED
|
@@ -0,0 +1,141 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
!
|
| 2 |
+
"
|
| 3 |
+
%
|
| 4 |
+
'
|
| 5 |
+
(
|
| 6 |
+
)
|
| 7 |
+
+
|
| 8 |
+
,
|
| 9 |
+
-
|
| 10 |
+
.
|
| 11 |
+
/
|
| 12 |
+
0
|
| 13 |
+
1
|
| 14 |
+
2
|
| 15 |
+
3
|
| 16 |
+
7
|
| 17 |
+
:
|
| 18 |
+
;
|
| 19 |
+
=
|
| 20 |
+
?
|
| 21 |
+
A
|
| 22 |
+
C
|
| 23 |
+
D
|
| 24 |
+
N
|
| 25 |
+
O
|
| 26 |
+
[
|
| 27 |
+
]
|
| 28 |
+
a
|
| 29 |
+
b
|
| 30 |
+
c
|
| 31 |
+
d
|
| 32 |
+
e
|
| 33 |
+
f
|
| 34 |
+
g
|
| 35 |
+
h
|
| 36 |
+
i
|
| 37 |
+
j
|
| 38 |
+
k
|
| 39 |
+
l
|
| 40 |
+
m
|
| 41 |
+
n
|
| 42 |
+
o
|
| 43 |
+
p
|
| 44 |
+
r
|
| 45 |
+
s
|
| 46 |
+
t
|
| 47 |
+
u
|
| 48 |
+
v
|
| 49 |
+
w
|
| 50 |
+
y
|
| 51 |
+
|
|
| 52 |
+
¥
|
| 53 |
+
÷
|
| 54 |
+
ʻ
|
| 55 |
+
ँ
|
| 56 |
+
ं
|
| 57 |
+
ः
|
| 58 |
+
अ
|
| 59 |
+
आ
|
| 60 |
+
इ
|
| 61 |
+
ई
|
| 62 |
+
उ
|
| 63 |
+
ऊ
|
| 64 |
+
ऋ
|
| 65 |
+
ऎ
|
| 66 |
+
ए
|
| 67 |
+
ऐ
|
| 68 |
+
ओ
|
| 69 |
+
औ
|
| 70 |
+
क
|
| 71 |
+
ख
|
| 72 |
+
ग
|
| 73 |
+
घ
|
| 74 |
+
ङ
|
| 75 |
+
च
|
| 76 |
+
छ
|
| 77 |
+
ज
|
| 78 |
+
झ
|
| 79 |
+
ञ
|
| 80 |
+
ट
|
| 81 |
+
ठ
|
| 82 |
+
ड
|
| 83 |
+
ढ
|
| 84 |
+
ण
|
| 85 |
+
त
|
| 86 |
+
थ
|
| 87 |
+
द
|
| 88 |
+
ध
|
| 89 |
+
न
|
| 90 |
+
ऩ
|
| 91 |
+
प
|
| 92 |
+
फ
|
| 93 |
+
ब
|
| 94 |
+
भ
|
| 95 |
+
म
|
| 96 |
+
य
|
| 97 |
+
र
|
| 98 |
+
ल
|
| 99 |
+
व
|
| 100 |
+
श
|
| 101 |
+
ष
|
| 102 |
+
स
|
| 103 |
+
ह
|
| 104 |
+
़
|
| 105 |
+
ऽ
|
| 106 |
+
ा
|
| 107 |
+
ि
|
| 108 |
+
ी
|
| 109 |
+
ु
|
| 110 |
+
ू
|
| 111 |
+
ृ
|
| 112 |
+
ॆ
|
| 113 |
+
े
|
| 114 |
+
ै
|
| 115 |
+
ो
|
| 116 |
+
ौ
|
| 117 |
+
्
|
| 118 |
+
।
|
| 119 |
+
०
|
| 120 |
+
१
|
| 121 |
+
२
|
| 122 |
+
३
|
| 123 |
+
४
|
| 124 |
+
५
|
| 125 |
+
६
|
| 126 |
+
७
|
| 127 |
+
८
|
| 128 |
+
९
|
| 129 |
+
ॽ
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
–
|
| 133 |
+
—
|
| 134 |
+
‘
|
| 135 |
+
’
|
| 136 |
+
‚
|
| 137 |
+
“
|
| 138 |
+
”
|
| 139 |
+
…
|
| 140 |
+
√
|
| 141 |
+
∶
|
ocr-tech-mps-integration/data/dictionaries/gurungkhema-v1.txt
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
|
ocr-tech-mps-integration/data/dictionaries/kiratrai-v1.txt
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
|
ocr-tech-mps-integration/data/dictionaries/lepcha-v1.txt
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
ᰀ
|
| 2 |
+
ᰁ
|
| 3 |
+
ᰂ
|
| 4 |
+
ᰃ
|
| 5 |
+
ᰄ
|
| 6 |
+
ᰅ
|
| 7 |
+
ᰆ
|
| 8 |
+
ᰇ
|
| 9 |
+
ᰈ
|
| 10 |
+
ᰉ
|
| 11 |
+
ᰊ
|
| 12 |
+
ᰋ
|
| 13 |
+
ᰌ
|
| 14 |
+
ᰍ
|
| 15 |
+
ᰎ
|
| 16 |
+
ᰏ
|
| 17 |
+
ᰐ
|
| 18 |
+
ᰑ
|
| 19 |
+
ᰒ
|
| 20 |
+
ᰓ
|
| 21 |
+
ᰔ
|
| 22 |
+
ᰕ
|
| 23 |
+
ᰖ
|
| 24 |
+
ᰗ
|
| 25 |
+
ᰘ
|
| 26 |
+
ᰙ
|
| 27 |
+
ᰚ
|
| 28 |
+
ᰛ
|
| 29 |
+
ᰜ
|
| 30 |
+
ᰝ
|
| 31 |
+
ᰞ
|
| 32 |
+
ᰟ
|
| 33 |
+
ᰠ
|
| 34 |
+
ᰡ
|
| 35 |
+
ᰢ
|
| 36 |
+
ᰣ
|
| 37 |
+
ᰤ
|
| 38 |
+
ᰥ
|
| 39 |
+
ᰦ
|
| 40 |
+
ᰧ
|
| 41 |
+
ᰨ
|
| 42 |
+
ᰩ
|
| 43 |
+
ᰪ
|
| 44 |
+
ᰫ
|
| 45 |
+
ᰬ
|
| 46 |
+
ᰭ
|
| 47 |
+
ᰮ
|
| 48 |
+
ᰯ
|
| 49 |
+
ᰰ
|
| 50 |
+
ᰱ
|
| 51 |
+
ᰲ
|
| 52 |
+
ᰳ
|
| 53 |
+
ᰴ
|
| 54 |
+
ᰵ
|
| 55 |
+
ᰶ
|
| 56 |
+
᰷
|
| 57 |
+
᰻
|
| 58 |
+
᰼
|
| 59 |
+
᰽
|
| 60 |
+
᰾
|
| 61 |
+
᰿
|
| 62 |
+
᱀
|
| 63 |
+
᱁
|
| 64 |
+
᱂
|
| 65 |
+
᱃
|
| 66 |
+
᱄
|
| 67 |
+
᱅
|
| 68 |
+
᱆
|
| 69 |
+
᱇
|
| 70 |
+
᱈
|
| 71 |
+
᱉
|
| 72 |
+
ᱍ
|
| 73 |
+
ᱎ
|
| 74 |
+
ᱏ
|
ocr-tech-mps-integration/data/dictionaries/limbu-sirijonga-v1.txt
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
)
|
| 2 |
+
,
|
| 3 |
+
-
|
| 4 |
+
.
|
| 5 |
+
:
|
| 6 |
+
|
|
| 7 |
+
।
|
| 8 |
+
॥
|
| 9 |
+
ᤀ
|
| 10 |
+
ᤁ
|
| 11 |
+
ᤂ
|
| 12 |
+
ᤃ
|
| 13 |
+
ᤄ
|
| 14 |
+
ᤅ
|
| 15 |
+
ᤆ
|
| 16 |
+
ᤇ
|
| 17 |
+
ᤈ
|
| 18 |
+
ᤉ
|
| 19 |
+
ᤊ
|
| 20 |
+
ᤋ
|
| 21 |
+
ᤌ
|
| 22 |
+
ᤍ
|
| 23 |
+
ᤎ
|
| 24 |
+
ᤏ
|
| 25 |
+
ᤐ
|
| 26 |
+
ᤑ
|
| 27 |
+
ᤒ
|
| 28 |
+
ᤓ
|
| 29 |
+
ᤔ
|
| 30 |
+
ᤕ
|
| 31 |
+
ᤖ
|
| 32 |
+
ᤗ
|
| 33 |
+
ᤘ
|
| 34 |
+
ᤙ
|
| 35 |
+
ᤚ
|
| 36 |
+
ᤛ
|
| 37 |
+
ᤜ
|
| 38 |
+
ᤝ
|
| 39 |
+
ᤞ
|
| 40 |
+
ᤠ
|
| 41 |
+
ᤡ
|
| 42 |
+
ᤢ
|
| 43 |
+
ᤣ
|
| 44 |
+
ᤤ
|
| 45 |
+
ᤥ
|
| 46 |
+
ᤦ
|
| 47 |
+
ᤧ
|
| 48 |
+
ᤨ
|
| 49 |
+
ᤩ
|
| 50 |
+
ᤪ
|
| 51 |
+
ᤫ
|
| 52 |
+
ᤰ
|
| 53 |
+
ᤱ
|
| 54 |
+
ᤲ
|
| 55 |
+
ᤳ
|
| 56 |
+
ᤴ
|
| 57 |
+
ᤵ
|
| 58 |
+
ᤶ
|
| 59 |
+
ᤷ
|
| 60 |
+
ᤸ
|
| 61 |
+
᤹
|
| 62 |
+
᤺
|
| 63 |
+
᤻
|
| 64 |
+
᥀
|
| 65 |
+
᥄
|
| 66 |
+
᥅
|
| 67 |
+
᥆
|
| 68 |
+
᥇
|
| 69 |
+
᥈
|
| 70 |
+
᥉
|
| 71 |
+
᥊
|
| 72 |
+
᥋
|
| 73 |
+
᥌
|
| 74 |
+
᥍
|
| 75 |
+
᥎
|
| 76 |
+
᥏
|
ocr-tech-mps-integration/data/dictionaries/limbu-sirijonga-v2.txt
ADDED
|
@@ -0,0 +1,91 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
!
|
| 2 |
+
"
|
| 3 |
+
'
|
| 4 |
+
(
|
| 5 |
+
)
|
| 6 |
+
*
|
| 7 |
+
+
|
| 8 |
+
,
|
| 9 |
+
-
|
| 10 |
+
.
|
| 11 |
+
/
|
| 12 |
+
:
|
| 13 |
+
=
|
| 14 |
+
?
|
| 15 |
+
S
|
| 16 |
+
X
|
| 17 |
+
[
|
| 18 |
+
]
|
| 19 |
+
a
|
| 20 |
+
c
|
| 21 |
+
i
|
| 22 |
+
s
|
| 23 |
+
t
|
| 24 |
+
|
|
| 25 |
+
।
|
| 26 |
+
॥
|
| 27 |
+
ᤀ
|
| 28 |
+
ᤁ
|
| 29 |
+
ᤂ
|
| 30 |
+
ᤃ
|
| 31 |
+
ᤄ
|
| 32 |
+
ᤅ
|
| 33 |
+
ᤆ
|
| 34 |
+
ᤇ
|
| 35 |
+
ᤈ
|
| 36 |
+
ᤋ
|
| 37 |
+
ᤌ
|
| 38 |
+
ᤍ
|
| 39 |
+
ᤎ
|
| 40 |
+
ᤏ
|
| 41 |
+
ᤐ
|
| 42 |
+
ᤑ
|
| 43 |
+
ᤒ
|
| 44 |
+
ᤓ
|
| 45 |
+
ᤔ
|
| 46 |
+
ᤕ
|
| 47 |
+
ᤖ
|
| 48 |
+
ᤗ
|
| 49 |
+
ᤘ
|
| 50 |
+
ᤙ
|
| 51 |
+
ᤛ
|
| 52 |
+
ᤜ
|
| 53 |
+
ᤠ
|
| 54 |
+
ᤡ
|
| 55 |
+
ᤢ
|
| 56 |
+
ᤣ
|
| 57 |
+
ᤤ
|
| 58 |
+
ᤥ
|
| 59 |
+
ᤦ
|
| 60 |
+
ᤧ
|
| 61 |
+
ᤨ
|
| 62 |
+
ᤩ
|
| 63 |
+
ᤪ
|
| 64 |
+
ᤫ
|
| 65 |
+
ᤰ
|
| 66 |
+
ᤱ
|
| 67 |
+
ᤳ
|
| 68 |
+
ᤴ
|
| 69 |
+
ᤵ
|
| 70 |
+
ᤶ
|
| 71 |
+
ᤷ
|
| 72 |
+
ᤸ
|
| 73 |
+
᤹
|
| 74 |
+
᤺
|
| 75 |
+
᤻
|
| 76 |
+
᥄
|
| 77 |
+
᥅
|
| 78 |
+
᥆
|
| 79 |
+
᥇
|
| 80 |
+
᥈
|
| 81 |
+
᥉
|
| 82 |
+
᥊
|
| 83 |
+
᥋
|
| 84 |
+
᥌
|
| 85 |
+
᥍
|
| 86 |
+
᥎
|
| 87 |
+
᥏
|
| 88 |
+
|
| 89 |
+
‘
|
| 90 |
+
’
|
| 91 |
+
…
|
ocr-tech-mps-integration/data/dictionaries/limbu-sirijonga-v3.txt
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
!
|
| 2 |
+
"
|
| 3 |
+
'
|
| 4 |
+
(
|
| 5 |
+
)
|
| 6 |
+
*
|
| 7 |
+
+
|
| 8 |
+
,
|
| 9 |
+
-
|
| 10 |
+
.
|
| 11 |
+
/
|
| 12 |
+
:
|
| 13 |
+
=
|
| 14 |
+
?
|
| 15 |
+
S
|
| 16 |
+
X
|
| 17 |
+
[
|
| 18 |
+
]
|
| 19 |
+
a
|
| 20 |
+
c
|
| 21 |
+
i
|
| 22 |
+
s
|
| 23 |
+
t
|
| 24 |
+
|
|
| 25 |
+
।
|
| 26 |
+
॥
|
| 27 |
+
ᤀ
|
| 28 |
+
ᤁ
|
| 29 |
+
ᤂ
|
| 30 |
+
ᤃ
|
| 31 |
+
ᤄ
|
| 32 |
+
ᤅ
|
| 33 |
+
ᤆ
|
| 34 |
+
ᤇ
|
| 35 |
+
ᤈ
|
| 36 |
+
ᤋ
|
| 37 |
+
ᤌ
|
| 38 |
+
ᤍ
|
| 39 |
+
ᤎ
|
| 40 |
+
ᤏ
|
| 41 |
+
ᤐ
|
| 42 |
+
ᤑ
|
| 43 |
+
ᤒ
|
| 44 |
+
ᤓ
|
| 45 |
+
ᤔ
|
| 46 |
+
ᤕ
|
| 47 |
+
ᤖ
|
| 48 |
+
ᤗ
|
| 49 |
+
ᤘ
|
| 50 |
+
ᤙ
|
| 51 |
+
ᤛ
|
| 52 |
+
ᤜ
|
| 53 |
+
ᤠ
|
| 54 |
+
ᤡ
|
| 55 |
+
ᤢ
|
| 56 |
+
ᤣ
|
| 57 |
+
ᤤ
|
| 58 |
+
ᤥ
|
| 59 |
+
ᤦ
|
| 60 |
+
ᤧ
|
| 61 |
+
ᤨ
|
| 62 |
+
ᤩ
|
| 63 |
+
ᤪ
|
| 64 |
+
ᤫ
|
| 65 |
+
ᤰ
|
| 66 |
+
ᤱ
|
| 67 |
+
ᤳ
|
| 68 |
+
ᤴ
|
| 69 |
+
ᤵ
|
| 70 |
+
ᤶ
|
| 71 |
+
ᤷ
|
| 72 |
+
ᤸ
|
| 73 |
+
᤹
|
| 74 |
+
᤺
|
| 75 |
+
᤻
|
| 76 |
+
᥄
|
| 77 |
+
᥅
|
| 78 |
+
᥆
|
| 79 |
+
᥇
|
| 80 |
+
᥈
|
| 81 |
+
᥉
|
| 82 |
+
᥊
|
| 83 |
+
᥋
|
| 84 |
+
᥌
|
| 85 |
+
᥍
|
| 86 |
+
᥎
|
| 87 |
+
᥏
|
| 88 |
+
|
| 89 |
+
‘
|
| 90 |
+
’
|
| 91 |
+
…
|
| 92 |
+
ᤉ
|
| 93 |
+
ᤊ
|
| 94 |
+
ᤚ
|
| 95 |
+
ᤝ
|
| 96 |
+
ᤞ
|
| 97 |
+
ᤲ
|
| 98 |
+
᥀
|
ocr-tech-mps-integration/data/dictionaries/newa-clean-v1.txt
ADDED
|
@@ -0,0 +1,103 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
!
|
| 2 |
+
%
|
| 3 |
+
(
|
| 4 |
+
)
|
| 5 |
+
+
|
| 6 |
+
,
|
| 7 |
+
-
|
| 8 |
+
.
|
| 9 |
+
/
|
| 10 |
+
0
|
| 11 |
+
1
|
| 12 |
+
2
|
| 13 |
+
6
|
| 14 |
+
8
|
| 15 |
+
9
|
| 16 |
+
:
|
| 17 |
+
;
|
| 18 |
+
?
|
| 19 |
+
~
|
| 20 |
+
|
| 21 |
+
¯
|
| 22 |
+
|
| 23 |
+
–
|
| 24 |
+
‘
|
| 25 |
+
’
|
| 26 |
+
“
|
| 27 |
+
”
|
| 28 |
+
•
|
| 29 |
+
𑐀
|
| 30 |
+
𑐁
|
| 31 |
+
𑐂
|
| 32 |
+
𑐃
|
| 33 |
+
𑐄
|
| 34 |
+
𑐅
|
| 35 |
+
𑐆
|
| 36 |
+
𑐈
|
| 37 |
+
𑐊
|
| 38 |
+
𑐌
|
| 39 |
+
𑐍
|
| 40 |
+
𑐎
|
| 41 |
+
𑐏
|
| 42 |
+
𑐐
|
| 43 |
+
𑐑
|
| 44 |
+
𑐒
|
| 45 |
+
𑐔
|
| 46 |
+
𑐕
|
| 47 |
+
𑐖
|
| 48 |
+
𑐗
|
| 49 |
+
𑐘
|
| 50 |
+
𑐚
|
| 51 |
+
𑐛
|
| 52 |
+
𑐜
|
| 53 |
+
𑐝
|
| 54 |
+
𑐞
|
| 55 |
+
𑐟
|
| 56 |
+
𑐠
|
| 57 |
+
𑐡
|
| 58 |
+
𑐢
|
| 59 |
+
𑐣
|
| 60 |
+
𑐤
|
| 61 |
+
𑐥
|
| 62 |
+
𑐦
|
| 63 |
+
𑐧
|
| 64 |
+
𑐨
|
| 65 |
+
𑐩
|
| 66 |
+
𑐫
|
| 67 |
+
𑐬
|
| 68 |
+
𑐮
|
| 69 |
+
𑐰
|
| 70 |
+
𑐱
|
| 71 |
+
𑐲
|
| 72 |
+
𑐳
|
| 73 |
+
𑐴
|
| 74 |
+
𑐵
|
| 75 |
+
𑐶
|
| 76 |
+
𑐷
|
| 77 |
+
𑐸
|
| 78 |
+
𑐹
|
| 79 |
+
𑐺
|
| 80 |
+
𑐻
|
| 81 |
+
𑐾
|
| 82 |
+
𑐿
|
| 83 |
+
𑑀
|
| 84 |
+
𑑁
|
| 85 |
+
𑑂
|
| 86 |
+
𑑃
|
| 87 |
+
𑑄
|
| 88 |
+
𑑅
|
| 89 |
+
𑑆
|
| 90 |
+
𑑇
|
| 91 |
+
𑑋
|
| 92 |
+
𑑌
|
| 93 |
+
𑑍
|
| 94 |
+
𑑐
|
| 95 |
+
𑑑
|
| 96 |
+
𑑒
|
| 97 |
+
𑑓
|
| 98 |
+
𑑔
|
| 99 |
+
𑑕
|
| 100 |
+
𑑖
|
| 101 |
+
𑑗
|
| 102 |
+
𑑘
|
| 103 |
+
𑑙
|
ocr-tech-mps-integration/data/dictionaries/newa-htr-v1.txt
ADDED
|
@@ -0,0 +1,177 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
|
| 3 |
+
!
|
| 4 |
+
%
|
| 5 |
+
(
|
| 6 |
+
)
|
| 7 |
+
+
|
| 8 |
+
,
|
| 9 |
+
-
|
| 10 |
+
.
|
| 11 |
+
/
|
| 12 |
+
0
|
| 13 |
+
1
|
| 14 |
+
2
|
| 15 |
+
3
|
| 16 |
+
4
|
| 17 |
+
6
|
| 18 |
+
7
|
| 19 |
+
8
|
| 20 |
+
9
|
| 21 |
+
:
|
| 22 |
+
;
|
| 23 |
+
?
|
| 24 |
+
A
|
| 25 |
+
B
|
| 26 |
+
C
|
| 27 |
+
D
|
| 28 |
+
E
|
| 29 |
+
F
|
| 30 |
+
G
|
| 31 |
+
H
|
| 32 |
+
I
|
| 33 |
+
J
|
| 34 |
+
L
|
| 35 |
+
M
|
| 36 |
+
N
|
| 37 |
+
O
|
| 38 |
+
P
|
| 39 |
+
R
|
| 40 |
+
S
|
| 41 |
+
U
|
| 42 |
+
W
|
| 43 |
+
Y
|
| 44 |
+
[
|
| 45 |
+
]
|
| 46 |
+
a
|
| 47 |
+
b
|
| 48 |
+
c
|
| 49 |
+
d
|
| 50 |
+
e
|
| 51 |
+
f
|
| 52 |
+
g
|
| 53 |
+
h
|
| 54 |
+
i
|
| 55 |
+
j
|
| 56 |
+
k
|
| 57 |
+
l
|
| 58 |
+
m
|
| 59 |
+
n
|
| 60 |
+
o
|
| 61 |
+
p
|
| 62 |
+
r
|
| 63 |
+
s
|
| 64 |
+
t
|
| 65 |
+
u
|
| 66 |
+
v
|
| 67 |
+
w
|
| 68 |
+
y
|
| 69 |
+
z
|
| 70 |
+
~
|
| 71 |
+
|
| 72 |
+
¯
|
| 73 |
+
»
|
| 74 |
+
|
| 75 |
+
–
|
| 76 |
+
‘
|
| 77 |
+
’
|
| 78 |
+
“
|
| 79 |
+
”
|
| 80 |
+
•
|
| 81 |
+
𑐀
|
| 82 |
+
𑐁
|
| 83 |
+
𑐂
|
| 84 |
+
𑐃
|
| 85 |
+
𑐄
|
| 86 |
+
𑐅
|
| 87 |
+
𑐆
|
| 88 |
+
𑐇
|
| 89 |
+
𑐈
|
| 90 |
+
𑐉
|
| 91 |
+
𑐊
|
| 92 |
+
𑐋
|
| 93 |
+
𑐌
|
| 94 |
+
𑐍
|
| 95 |
+
𑐎
|
| 96 |
+
𑐏
|
| 97 |
+
𑐐
|
| 98 |
+
𑐑
|
| 99 |
+
𑐒
|
| 100 |
+
𑐓
|
| 101 |
+
𑐔
|
| 102 |
+
𑐕
|
| 103 |
+
𑐖
|
| 104 |
+
𑐗
|
| 105 |
+
𑐘
|
| 106 |
+
𑐙
|
| 107 |
+
𑐚
|
| 108 |
+
𑐛
|
| 109 |
+
𑐜
|
| 110 |
+
𑐝
|
| 111 |
+
𑐞
|
| 112 |
+
𑐟
|
| 113 |
+
𑐠
|
| 114 |
+
𑐡
|
| 115 |
+
𑐢
|
| 116 |
+
𑐣
|
| 117 |
+
𑐤
|
| 118 |
+
𑐥
|
| 119 |
+
𑐦
|
| 120 |
+
𑐧
|
| 121 |
+
𑐨
|
| 122 |
+
𑐩
|
| 123 |
+
𑐪
|
| 124 |
+
𑐫
|
| 125 |
+
𑐬
|
| 126 |
+
𑐭
|
| 127 |
+
𑐮
|
| 128 |
+
𑐯
|
| 129 |
+
𑐰
|
| 130 |
+
𑐱
|
| 131 |
+
𑐲
|
| 132 |
+
𑐳
|
| 133 |
+
𑐴
|
| 134 |
+
𑐵
|
| 135 |
+
𑐶
|
| 136 |
+
𑐷
|
| 137 |
+
𑐸
|
| 138 |
+
𑐹
|
| 139 |
+
𑐺
|
| 140 |
+
𑐻
|
| 141 |
+
𑐼
|
| 142 |
+
𑐽
|
| 143 |
+
𑐾
|
| 144 |
+
𑐿
|
| 145 |
+
𑑀
|
| 146 |
+
𑑁
|
| 147 |
+
𑑂
|
| 148 |
+
𑑃
|
| 149 |
+
𑑄
|
| 150 |
+
𑑅
|
| 151 |
+
𑑆
|
| 152 |
+
𑑇
|
| 153 |
+
𑑈
|
| 154 |
+
𑑉
|
| 155 |
+
𑑊
|
| 156 |
+
𑑋
|
| 157 |
+
𑑌
|
| 158 |
+
𑑍
|
| 159 |
+
𑑎
|
| 160 |
+
𑑏
|
| 161 |
+
𑑐
|
| 162 |
+
𑑑
|
| 163 |
+
𑑒
|
| 164 |
+
𑑓
|
| 165 |
+
𑑔
|
| 166 |
+
𑑕
|
| 167 |
+
𑑖
|
| 168 |
+
𑑗
|
| 169 |
+
𑑘
|
| 170 |
+
𑑙
|
| 171 |
+
𑑚
|
| 172 |
+
𑑛
|
| 173 |
+
𑑝
|
| 174 |
+
𑑞
|
| 175 |
+
𑑟
|
| 176 |
+
𑑠
|
| 177 |
+
𑑡
|
ocr-tech-mps-integration/data/dictionaries/newa-native-v1.txt
ADDED
|
@@ -0,0 +1,163 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
!
|
| 2 |
+
%
|
| 3 |
+
(
|
| 4 |
+
)
|
| 5 |
+
*
|
| 6 |
+
+
|
| 7 |
+
,
|
| 8 |
+
-
|
| 9 |
+
.
|
| 10 |
+
/
|
| 11 |
+
0
|
| 12 |
+
1
|
| 13 |
+
2
|
| 14 |
+
3
|
| 15 |
+
4
|
| 16 |
+
5
|
| 17 |
+
6
|
| 18 |
+
7
|
| 19 |
+
8
|
| 20 |
+
9
|
| 21 |
+
:
|
| 22 |
+
;
|
| 23 |
+
?
|
| 24 |
+
A
|
| 25 |
+
B
|
| 26 |
+
C
|
| 27 |
+
D
|
| 28 |
+
E
|
| 29 |
+
F
|
| 30 |
+
H
|
| 31 |
+
I
|
| 32 |
+
J
|
| 33 |
+
L
|
| 34 |
+
M
|
| 35 |
+
N
|
| 36 |
+
O
|
| 37 |
+
P
|
| 38 |
+
R
|
| 39 |
+
S
|
| 40 |
+
T
|
| 41 |
+
U
|
| 42 |
+
W
|
| 43 |
+
Y
|
| 44 |
+
[
|
| 45 |
+
]
|
| 46 |
+
_
|
| 47 |
+
a
|
| 48 |
+
b
|
| 49 |
+
c
|
| 50 |
+
d
|
| 51 |
+
e
|
| 52 |
+
f
|
| 53 |
+
g
|
| 54 |
+
h
|
| 55 |
+
i
|
| 56 |
+
k
|
| 57 |
+
l
|
| 58 |
+
m
|
| 59 |
+
n
|
| 60 |
+
o
|
| 61 |
+
p
|
| 62 |
+
r
|
| 63 |
+
s
|
| 64 |
+
t
|
| 65 |
+
u
|
| 66 |
+
v
|
| 67 |
+
w
|
| 68 |
+
y
|
| 69 |
+
z
|
| 70 |
+
~
|
| 71 |
+
¯
|
| 72 |
+
»
|
| 73 |
+
ə
|
| 74 |
+
ˈ
|
| 75 |
+
्
|
| 76 |
+
।
|
| 77 |
+
॥
|
| 78 |
+
॰
|
| 79 |
+
|
| 80 |
+
–
|
| 81 |
+
‘
|
| 82 |
+
’
|
| 83 |
+
“
|
| 84 |
+
”
|
| 85 |
+
•
|
| 86 |
+
…
|
| 87 |
+
・
|
| 88 |
+
𑐀
|
| 89 |
+
𑐁
|
| 90 |
+
𑐂
|
| 91 |
+
𑐃
|
| 92 |
+
𑐄
|
| 93 |
+
𑐅
|
| 94 |
+
𑐆
|
| 95 |
+
𑐈
|
| 96 |
+
𑐊
|
| 97 |
+
𑐌
|
| 98 |
+
𑐍
|
| 99 |
+
𑐎
|
| 100 |
+
𑐏
|
| 101 |
+
𑐐
|
| 102 |
+
𑐑
|
| 103 |
+
𑐒
|
| 104 |
+
𑐔
|
| 105 |
+
𑐕
|
| 106 |
+
𑐖
|
| 107 |
+
𑐗
|
| 108 |
+
𑐘
|
| 109 |
+
𑐚
|
| 110 |
+
𑐛
|
| 111 |
+
𑐜
|
| 112 |
+
𑐝
|
| 113 |
+
𑐞
|
| 114 |
+
𑐟
|
| 115 |
+
𑐠
|
| 116 |
+
𑐡
|
| 117 |
+
𑐢
|
| 118 |
+
𑐣
|
| 119 |
+
𑐤
|
| 120 |
+
𑐥
|
| 121 |
+
𑐦
|
| 122 |
+
𑐧
|
| 123 |
+
𑐨
|
| 124 |
+
𑐩
|
| 125 |
+
𑐫
|
| 126 |
+
𑐬
|
| 127 |
+
𑐮
|
| 128 |
+
𑐰
|
| 129 |
+
𑐱
|
| 130 |
+
𑐲
|
| 131 |
+
𑐳
|
| 132 |
+
𑐴
|
| 133 |
+
𑐵
|
| 134 |
+
𑐶
|
| 135 |
+
𑐷
|
| 136 |
+
𑐸
|
| 137 |
+
𑐹
|
| 138 |
+
𑐺
|
| 139 |
+
𑐻
|
| 140 |
+
𑐾
|
| 141 |
+
𑐿
|
| 142 |
+
𑑀
|
| 143 |
+
𑑁
|
| 144 |
+
𑑂
|
| 145 |
+
𑑃
|
| 146 |
+
𑑄
|
| 147 |
+
𑑅
|
| 148 |
+
𑑆
|
| 149 |
+
𑑇
|
| 150 |
+
𑑉
|
| 151 |
+
𑑋
|
| 152 |
+
𑑌
|
| 153 |
+
𑑍
|
| 154 |
+
𑑐
|
| 155 |
+
𑑑
|
| 156 |
+
𑑒
|
| 157 |
+
𑑓
|
| 158 |
+
𑑔
|
| 159 |
+
𑑕
|
| 160 |
+
𑑖
|
| 161 |
+
𑑗
|
| 162 |
+
𑑘
|
| 163 |
+
𑑙
|
ocr-tech-mps-integration/data/dictionaries/newa-v1.txt
ADDED
|
@@ -0,0 +1,97 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
𑐀
|
| 2 |
+
𑐁
|
| 3 |
+
𑐂
|
| 4 |
+
𑐃
|
| 5 |
+
𑐄
|
| 6 |
+
𑐅
|
| 7 |
+
𑐆
|
| 8 |
+
𑐇
|
| 9 |
+
𑐈
|
| 10 |
+
𑐉
|
| 11 |
+
𑐊
|
| 12 |
+
𑐋
|
| 13 |
+
𑐌
|
| 14 |
+
𑐍
|
| 15 |
+
𑐎
|
| 16 |
+
𑐏
|
| 17 |
+
𑐐
|
| 18 |
+
𑐑
|
| 19 |
+
𑐒
|
| 20 |
+
𑐓
|
| 21 |
+
𑐔
|
| 22 |
+
𑐕
|
| 23 |
+
𑐖
|
| 24 |
+
𑐗
|
| 25 |
+
𑐘
|
| 26 |
+
𑐙
|
| 27 |
+
𑐚
|
| 28 |
+
𑐛
|
| 29 |
+
𑐜
|
| 30 |
+
𑐝
|
| 31 |
+
𑐞
|
| 32 |
+
𑐟
|
| 33 |
+
𑐠
|
| 34 |
+
𑐡
|
| 35 |
+
𑐢
|
| 36 |
+
𑐣
|
| 37 |
+
𑐤
|
| 38 |
+
𑐥
|
| 39 |
+
𑐦
|
| 40 |
+
𑐧
|
| 41 |
+
𑐨
|
| 42 |
+
𑐩
|
| 43 |
+
𑐪
|
| 44 |
+
𑐫
|
| 45 |
+
𑐬
|
| 46 |
+
𑐭
|
| 47 |
+
𑐮
|
| 48 |
+
𑐯
|
| 49 |
+
𑐰
|
| 50 |
+
𑐱
|
| 51 |
+
𑐲
|
| 52 |
+
𑐳
|
| 53 |
+
𑐴
|
| 54 |
+
𑐵
|
| 55 |
+
𑐶
|
| 56 |
+
𑐷
|
| 57 |
+
𑐸
|
| 58 |
+
𑐹
|
| 59 |
+
𑐺
|
| 60 |
+
𑐻
|
| 61 |
+
𑐼
|
| 62 |
+
𑐽
|
| 63 |
+
𑐾
|
| 64 |
+
𑐿
|
| 65 |
+
𑑀
|
| 66 |
+
𑑁
|
| 67 |
+
𑑂
|
| 68 |
+
𑑃
|
| 69 |
+
𑑄
|
| 70 |
+
𑑅
|
| 71 |
+
𑑆
|
| 72 |
+
𑑇
|
| 73 |
+
𑑈
|
| 74 |
+
𑑉
|
| 75 |
+
𑑊
|
| 76 |
+
𑑋
|
| 77 |
+
𑑌
|
| 78 |
+
𑑍
|
| 79 |
+
𑑎
|
| 80 |
+
𑑏
|
| 81 |
+
𑑐
|
| 82 |
+
𑑑
|
| 83 |
+
𑑒
|
| 84 |
+
𑑓
|
| 85 |
+
𑑔
|
| 86 |
+
𑑕
|
| 87 |
+
𑑖
|
| 88 |
+
𑑗
|
| 89 |
+
𑑘
|
| 90 |
+
𑑙
|
| 91 |
+
𑑚
|
| 92 |
+
𑑛
|
| 93 |
+
𑑝
|
| 94 |
+
𑑞
|
| 95 |
+
𑑟
|
| 96 |
+
𑑠
|
| 97 |
+
𑑡
|
ocr-tech-mps-integration/data/dictionaries/sunuwar-v1.txt
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
|
ocr-tech-mps-integration/data/dictionaries/tirhuta-v1.txt
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
𑒀
|
| 2 |
+
𑒁
|
| 3 |
+
𑒂
|
| 4 |
+
𑒃
|
| 5 |
+
𑒄
|
| 6 |
+
𑒅
|
| 7 |
+
𑒆
|
| 8 |
+
𑒇
|
| 9 |
+
𑒈
|
| 10 |
+
𑒉
|
| 11 |
+
𑒊
|
| 12 |
+
𑒋
|
| 13 |
+
𑒌
|
| 14 |
+
𑒍
|
| 15 |
+
𑒎
|
| 16 |
+
𑒏
|
| 17 |
+
𑒐
|
| 18 |
+
𑒑
|
| 19 |
+
𑒒
|
| 20 |
+
𑒓
|
| 21 |
+
𑒔
|
| 22 |
+
𑒕
|
| 23 |
+
𑒖
|
| 24 |
+
𑒗
|
| 25 |
+
𑒘
|
| 26 |
+
𑒙
|
| 27 |
+
𑒚
|
| 28 |
+
𑒛
|
| 29 |
+
𑒜
|
| 30 |
+
𑒝
|
| 31 |
+
𑒞
|
| 32 |
+
𑒟
|
| 33 |
+
𑒠
|
| 34 |
+
𑒡
|
| 35 |
+
𑒢
|
| 36 |
+
𑒣
|
| 37 |
+
𑒤
|
| 38 |
+
𑒥
|
| 39 |
+
𑒦
|
| 40 |
+
𑒧
|
| 41 |
+
𑒨
|
| 42 |
+
𑒩
|
| 43 |
+
𑒪
|
| 44 |
+
𑒫
|
| 45 |
+
𑒬
|
| 46 |
+
𑒭
|
| 47 |
+
𑒮
|
| 48 |
+
𑒯
|
| 49 |
+
𑒰
|
| 50 |
+
𑒱
|
| 51 |
+
𑒲
|
| 52 |
+
𑒳
|
| 53 |
+
𑒴
|
| 54 |
+
𑒵
|
| 55 |
+
𑒶
|
| 56 |
+
𑒷
|
| 57 |
+
𑒸
|
| 58 |
+
𑒹
|
| 59 |
+
𑒺
|
| 60 |
+
𑒻
|
| 61 |
+
𑒼
|
| 62 |
+
𑒽
|
| 63 |
+
𑒾
|
| 64 |
+
𑒿
|
| 65 |
+
𑓀
|
| 66 |
+
𑓁
|
| 67 |
+
𑓂
|
| 68 |
+
𑓃
|
| 69 |
+
𑓄
|
| 70 |
+
𑓅
|
| 71 |
+
𑓆
|
| 72 |
+
𑓇
|
| 73 |
+
𑓐
|
| 74 |
+
𑓑
|
| 75 |
+
𑓒
|
| 76 |
+
𑓓
|
| 77 |
+
𑓔
|
| 78 |
+
𑓕
|
| 79 |
+
𑓖
|
| 80 |
+
𑓗
|
| 81 |
+
𑓘
|
| 82 |
+
𑓙
|
ocr-tech-mps-integration/data/mappings/limbu-legacy/Limbu.map
ADDED
|
@@ -0,0 +1,146 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
; Limbu.map
|
| 2 |
+
; Copyright (c) 2007 SIL International.
|
| 3 |
+
|
| 4 |
+
; TECkit mapping to convert from Limbu legacy encoding (as used by Sirijonga)
|
| 5 |
+
; to Unicode and back
|
| 6 |
+
|
| 7 |
+
EncodingName "Limbu-Sirijonga"
|
| 8 |
+
DescriptiveName "Limbu"
|
| 9 |
+
Version "1.0"
|
| 10 |
+
Contact "mailto:victor_gaultney@sil.org"
|
| 11 |
+
RegistrationAuthority "SIL International"
|
| 12 |
+
RegistrationName "Limbu-001"
|
| 13 |
+
Copyright "(c)2007 SIL International"
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
Pass(Byte_Unicode)
|
| 17 |
+
|
| 18 |
+
ByteDefault 0x5E ; number sign
|
| 19 |
+
UniDefault replacement_character
|
| 20 |
+
|
| 21 |
+
ByteClass [CTL] = ( 0x00 .. 0x1F )
|
| 22 |
+
UniClass [CTL] = ( U+0000 .. U+001F )
|
| 23 |
+
[CTL] <> [CTL]
|
| 24 |
+
|
| 25 |
+
; Unicode chars with no glyphs in legacy font:
|
| 26 |
+
; U+1909 (LIMBU LETTER JHA)
|
| 27 |
+
; U+190A (LIMBU LETTER YAN)
|
| 28 |
+
; U+191A (LIMBU LETTER SSA)
|
| 29 |
+
; U+1932 (LIMBU SMALL LETTER ANUSVARA)
|
| 30 |
+
; U+1940 (LIMBU SIGN LOO)
|
| 31 |
+
; U+1944 (LIMBU EXCLAMATION MARK)
|
| 32 |
+
|
| 33 |
+
;mappings - * means direct from ASCII, ** means to another Unicode block
|
| 34 |
+
0x20 <> U+0020 ; space*
|
| 35 |
+
0x21 <> U+0021 ; exclam*
|
| 36 |
+
0x22 <> U+0022 ; quotedbl*
|
| 37 |
+
;0x23 <> ?
|
| 38 |
+
0x24 <> U+0024 ; dollar* (NOT IN FONT)
|
| 39 |
+
0x25 <> U+0025 ; percent*
|
| 40 |
+
0x26 <> U+0026 ; ampersand* (NOT IN FONT)
|
| 41 |
+
0x27 <> U+1922 ; limbu_vowel_sign_u
|
| 42 |
+
0x28 <> U+0028 ; parenleft*
|
| 43 |
+
0x29 <> U+0029 ; parenright*
|
| 44 |
+
0x2A <> U+002A ; asterisk*
|
| 45 |
+
0x2B <> U+002B ; plus*
|
| 46 |
+
0x2C <> U+002C ; comma*
|
| 47 |
+
0x2D <> U+193B ; limbu_sign_sa-i
|
| 48 |
+
0x2E <> U+0965 ; devanagari_double_danda**
|
| 49 |
+
0x2F <> U+002F ; solidus*
|
| 50 |
+
0x30 <> U+1946 ; limbu_digit_zero
|
| 51 |
+
0x31 <> U+1947 ; limbu_digit_one
|
| 52 |
+
0x32 <> U+1948 ; limbu_digit_two
|
| 53 |
+
0x33 <> U+1949 ; limbu_digit_three
|
| 54 |
+
0x34 <> U+194A ; limbu_digit_four
|
| 55 |
+
0x35 <> U+194B ; limbu_digit_five
|
| 56 |
+
0x36 <> U+194C ; limbu_digit_six
|
| 57 |
+
0x37 <> U+194D ; limbu_digit_seven
|
| 58 |
+
0x38 <> U+194E ; limbu_digit_eight
|
| 59 |
+
0x39 <> U+194F ; limbu_digit_nine
|
| 60 |
+
0x3A > U+003A ; colon (0x60 is appropriate reverse mapping)
|
| 61 |
+
0x3B <> U+191B ; limbu_letter_sa
|
| 62 |
+
0x3C <> U+005B ; bracketleft
|
| 63 |
+
0x3D <> U+003D ; equal*
|
| 64 |
+
0x3E <> U+005D ; bracketright*
|
| 65 |
+
0x3F <> U+003F ; question*
|
| 66 |
+
0x40 <> U+0040 ; at*
|
| 67 |
+
0x41 <> U+0041 ; latin_capital_letter_A (NOT IN FONT)
|
| 68 |
+
0x42 <> U+003B ; semicolon
|
| 69 |
+
0x43 <> U+1939 ; limbu_sign_mukphreng
|
| 70 |
+
0x44 <> U+1936 ; limbu_small_letter_ma
|
| 71 |
+
0x45 <> U+007B ; braceleft
|
| 72 |
+
0x46 <> U+1916 ; limbu_letter_ra
|
| 73 |
+
0x47 <> U+1934 ; limbu_small_letter_na
|
| 74 |
+
;0x48 <> U+0048 ; latin_capital_letter_H (ORIGINAL)
|
| 75 |
+
0x48 <> U+1922 U+192A ; limbu_vowel_sign_u + limbu_subjoined_letter_ra (NEW)
|
| 76 |
+
0x49 <> U+1931 ; limbu_small_letter_nga
|
| 77 |
+
0x4A <> U+192B ; limbu_subjoined_letter_wa
|
| 78 |
+
0x4B <> U+1935 ; limbu_small_letter_pa
|
| 79 |
+
0x4C <> U+1921 U+193A ; limbu_vowel_sign_i + limbu_sign_kemphreng
|
| 80 |
+
0x4D <> U+193A ; limbu_sign_kemphreng
|
| 81 |
+
0x4E <> U+1938 ; limbu_small_letter_la
|
| 82 |
+
0x4F <> U+1929 ; limbu_subjoined_letter_ya
|
| 83 |
+
0x50 <> U+1904 ; limbu_letter_gha
|
| 84 |
+
0x51 <> U+003C ; less
|
| 85 |
+
0x52 <> U+007D ; braceright
|
| 86 |
+
0x53 <> U+1930 ; limbu_small_letter_ka
|
| 87 |
+
0x54 <> U+1933 ; limbu_small_letter_ta
|
| 88 |
+
0x55 <> U+1937 ; limbu_small_letter_ra
|
| 89 |
+
0x56 <> U+002E ; period
|
| 90 |
+
0x57 <> U+003E ; greater
|
| 91 |
+
;0x58 <> ? ; is this the same as the limbu exclamation mark?
|
| 92 |
+
0x59 <> U+0027 ; quotesingle
|
| 93 |
+
0x5A <> U+1945 ; limbu_question_mark
|
| 94 |
+
0x5B <> U+1927 ; limbu_vowel_sign_e
|
| 95 |
+
0x5C <> U+005C ; reverse_solidus*
|
| 96 |
+
0x5D <> U+1923 ; limbu_vowel_sign_ee (alone)
|
| 97 |
+
0x5E <> U+0023 ; numbersign
|
| 98 |
+
0x5F <> U+002D ; hyphen
|
| 99 |
+
0x60 <> U+003A ; colon
|
| 100 |
+
0x61 <> U+1912 ; limbu_letter_ba
|
| 101 |
+
0x62 <> U+190D ; limbu_letter_da
|
| 102 |
+
0x63 <> U+1900 ; limbu_vowel-carrier_letter
|
| 103 |
+
0x64 <> U+1914 ; limbu_letter_ma
|
| 104 |
+
0x65 <> U+1913 ; limbu_letter_bha
|
| 105 |
+
0x66 0x5D <> U+1925 ; limbu_vowel_sign_oo
|
| 106 |
+
0x66 0x7D <> U+1926 ; limbu_vowel_sign_au
|
| 107 |
+
0x66 <> U+1920 ; limbu_vowel_sign_a (alone)
|
| 108 |
+
0x67 <> U+190F ; limbu_letter_na
|
| 109 |
+
0x68 <> U+1908 ; limbu_letter_ja
|
| 110 |
+
0x69 <> U+1905 ; limbu_letter_nga
|
| 111 |
+
0x6A <> U+1918 ; limbu_letter_wa
|
| 112 |
+
0x6B <> U+1910 ; limbu_letter_pa
|
| 113 |
+
0x6C <> U+1921 ; limbu_vowel_sign_i
|
| 114 |
+
0x6D <> U+1911 ; limbu_letter_pha
|
| 115 |
+
0x6E <> U+1917 ; limbu_letter_la
|
| 116 |
+
0x6F <> U+1915 ; limbu_letter_ya
|
| 117 |
+
0x70 <> U+1907 ; limbu_letter_cha
|
| 118 |
+
0x71 <> U+192A ; limbu_subjoined_letter_ra
|
| 119 |
+
0x72 <> U+1906 ; limbu_letter_ca
|
| 120 |
+
0x73 <> U+1901 ; limbu_letter_ka
|
| 121 |
+
0x74 <> U+190B ; limbu_letter_ta
|
| 122 |
+
0x75 <> U+1903 ; limbu_letter_ga
|
| 123 |
+
0x76 <> U+1902 ; limbu_letter_kha
|
| 124 |
+
0x77 <> U+190E ; limbu_letter_dha
|
| 125 |
+
0x78 <> U+191C ; limbu_letter_ha
|
| 126 |
+
0x79 <> U+190C ; limbu_letter_tha
|
| 127 |
+
0x7A <> U+1919 ; limbu_letter_sha
|
| 128 |
+
0x7B <> U+1928 ; limbu_vowel_sign_o
|
| 129 |
+
0x7C <> U+007C ; bar*
|
| 130 |
+
0x7D <> U+1924 ; limbu_vowel_sign_ai (alone)
|
| 131 |
+
0x7E <> U+007E ; asciitilde*
|
| 132 |
+
;0x7F <> ;
|
| 133 |
+
|
| 134 |
+
0xD2 <> U+201C ; quotedblleft
|
| 135 |
+
0xD3 <> U+201D ; quotedblright
|
| 136 |
+
0xD4 <> U+2018 ; quotesingleleft
|
| 137 |
+
0xD5 <> U+2019 ; quotesingleright
|
| 138 |
+
|
| 139 |
+
Pass(Unicode)
|
| 140 |
+
; to reorder the subjoined cons
|
| 141 |
+
|
| 142 |
+
UniClass [VOWEL] = ( U+1920 .. U+1928 )
|
| 143 |
+
UniClass [SUBJ] = ( U+1929 .. U+192B )
|
| 144 |
+
|
| 145 |
+
[VOWEL]=V [SUBJ]=S <> @S @V
|
| 146 |
+
[VOWEL]=V U+193A [SUBJ]=S <> @S @V U+193A
|
ocr-tech-mps-integration/data/mappings/limbu-legacy/README.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
Compiled and uncompiled TECkit conversion tables are included here. The Limbu one maps data using the Namdhinggo SIL L custom-encoded font to Unicode.
|
ocr-tech-mps-integration/docs/ACTIVE_RESEARCH_CHECKLIST.md
ADDED
|
@@ -0,0 +1,386 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Active Research Checklist
|
| 2 |
+
|
| 3 |
+
This checklist records the current steering decision for the Nepal multilingual
|
| 4 |
+
OCR paper track. Keep it updated when remote data, synthesis converters,
|
| 5 |
+
validation packs, or claim gates change.
|
| 6 |
+
|
| 7 |
+
## Current Goal
|
| 8 |
+
|
| 9 |
+
Build a PaddleOCR-centered document OCR engine for at least ten languages
|
| 10 |
+
spoken in Nepal. Limbu/Sirijonga is the first non-Devanagari target. The
|
| 11 |
+
engine must support OCR, reading order, tables, figures, captions, Markdown,
|
| 12 |
+
and structured JSON. Synthetic data can train and ablate the system, but paper
|
| 13 |
+
claims require real held-out validation, especially Gorkhapatra/Naya Nepal
|
| 14 |
+
page evidence.
|
| 15 |
+
|
| 16 |
+
## Non-Negotiable Gates
|
| 17 |
+
|
| 18 |
+
- [ ] Keep PaddleOCR/PP-OCR as the primary engine family.
|
| 19 |
+
- [ ] Use PP-Structure-style document extraction for layout, reading order,
|
| 20 |
+
table reconstruction, figures, captions, Markdown, and JSON.
|
| 21 |
+
- [ ] Use PaddleOCR-VL or another VLM only as fallback, teacher, or hard-page
|
| 22 |
+
comparator; report fallback rate separately.
|
| 23 |
+
- [ ] Do not call a language synthesis-ready until a text manifest passes
|
| 24 |
+
`ocrtech audit-synthesis-text-manifest`.
|
| 25 |
+
- [ ] Do not call rendered OCR data training-ready until rendered outputs pass
|
| 26 |
+
`ocrtech audit-rendered-synthesis-lines`.
|
| 27 |
+
- [ ] For split synthesis text, render with `render-synthesis-text-split` so
|
| 28 |
+
train/eval labels and rendered audits are created together.
|
| 29 |
+
- [ ] Do not use synthetic samples as claim evidence.
|
| 30 |
+
- [ ] Do not promote Gorkhapatra/Naya Nepal pages to claim evidence until the
|
| 31 |
+
review audit and reference audit pass.
|
| 32 |
+
- [ ] Keep Nepali and English non-regression gates in every claim-grade run.
|
| 33 |
+
|
| 34 |
+
## cdjk Remote Synthesis Path
|
| 35 |
+
|
| 36 |
+
Remote host verified read-only on 2026-06-16:
|
| 37 |
+
|
| 38 |
+
```text
|
| 39 |
+
host: cdjk@100.110.148.43
|
| 40 |
+
machine: cdjks-MacBook-Pro.local
|
| 41 |
+
disk: /Volumes/TRANSCEND, 3.6Ti total, 2.0Ti available
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
Confirmed `~/github/llm` leads:
|
| 45 |
+
|
| 46 |
+
- `/Users/cdjk/github/llm/limbu-speech-toolkit`
|
| 47 |
+
- `/Users/cdjk/github/llm/rai-speech-toolkit`
|
| 48 |
+
- `/Users/cdjk/github/llm/dhimal-toolkit`
|
| 49 |
+
- `/Users/cdjk/github/llm/gurung-language-toolkit`
|
| 50 |
+
- `/Users/cdjk/github/llm/magar-toolkit`
|
| 51 |
+
- `/Users/cdjk/github/llm/ltk`
|
| 52 |
+
- `/Users/cdjk/github/llm/ltk/limbu`
|
| 53 |
+
- `/Users/cdjk/github/llm/ltk/tamang`
|
| 54 |
+
- `/Users/cdjk/github/llm/ltk/magar`
|
| 55 |
+
- `/Users/cdjk/github/llm/nepa-newa-text-frontend`
|
| 56 |
+
- `/Users/cdjk/github/llm/nepali_text_frontend`
|
| 57 |
+
- `/Users/cdjk/github/llm/devanagari-OCR`
|
| 58 |
+
- `/Users/cdjk/github/llm/nllb-better`
|
| 59 |
+
|
| 60 |
+
Confirmed `/Volumes/TRANSCEND` leads:
|
| 61 |
+
|
| 62 |
+
- `/Volumes/TRANSCEND/bible_brain_nepal/lif/LIFWBT_Limbu`
|
| 63 |
+
- `/Volumes/TRANSCEND/bible_brain_nepal/dhi/DHIDPI_Dhimal`
|
| 64 |
+
- `/Volumes/TRANSCEND/bible_brain_nepal/gvr/GVRWBT_Gurung`
|
| 65 |
+
- `/Volumes/TRANSCEND/bible_brain_nepal/new/NEWNCL_Newari`
|
| 66 |
+
- `/Volumes/TRANSCEND/bible_brain_nepal/taj/TAJWBT_Tamang__Eastern`
|
| 67 |
+
- `/Volumes/TRANSCEND/bible_brain_nepal/npi/NPIBIB_Nepali`
|
| 68 |
+
- `/Volumes/TRANSCEND/bible_brain_nepal/npi/NPIDPI_Nepali`
|
| 69 |
+
- `/Volumes/TRANSCEND/nepali_dataset_prep`
|
| 70 |
+
- `/Volumes/TRANSCEND/noiseremover_nepal18_timed_pipeline/bible_brain`
|
| 71 |
+
- `/Volumes/TRANSCEND/noiseremover_nepal18_timed_pipeline/speaker_cluster_limbu`
|
| 72 |
+
|
| 73 |
+
Next remote steps:
|
| 74 |
+
|
| 75 |
+
- [x] Selectively migrate text corpus/dictionary resources from
|
| 76 |
+
`cdjk@100.110.148.43` into ignored local `data/` without writing generated
|
| 77 |
+
corpus data back to the source machine. Tracked policy:
|
| 78 |
+
`configs/cdjk-language-corpus-migration-v1.json`.
|
| 79 |
+
- [x] Run `ocrtech audit-synthesis-resources` on the migrated roots:
|
| 80 |
+
`passed=True roots=6/6 files=2308 warnings=0`.
|
| 81 |
+
- [x] Prepare and audit first migrated synthesis-text manifests:
|
| 82 |
+
Limbu Limdic `26152` rows, Limbu-only subset `8772` rows, Tamang `46314`
|
| 83 |
+
rows, and Magar Western dictionary `1182` rows; all audits passed with
|
| 84 |
+
`claim_eligible=0`.
|
| 85 |
+
- [x] Render a corrected Limbu-only Namdhinggo smoke set with six degradation
|
| 86 |
+
profiles: `120` images, rendered audit `passed=True`, `issues=0`,
|
| 87 |
+
`warnings=0`.
|
| 88 |
+
- [ ] Convert the same local migration into full split train/eval synthesis
|
| 89 |
+
manifests with exclusion sets and license policy before large-scale
|
| 90 |
+
rendering.
|
| 91 |
+
- [ ] Inspect actual schemas for each candidate file before writing converters.
|
| 92 |
+
Do not infer fields from filenames.
|
| 93 |
+
- [x] Run explicit field-mapped conversion recipes for Gurung, Dhimal, and Rai
|
| 94 |
+
dialect corpora discovered in the migration.
|
| 95 |
+
- [x] Run explicit field-mapped conversion for extra Magar
|
| 96 |
+
`mgp/source/mscore` monolingual rows and `mgp/npi` lexicon/parallel rows.
|
| 97 |
+
- [x] Audit 20 migrated synthesis-text manifests after the second sweep:
|
| 98 |
+
`228212` total rows, `202560` Devanagari rows, `8772` true Unicode Limbu
|
| 99 |
+
rows, and all manifest audits passed with `claim_eligible=0`.
|
| 100 |
+
- [ ] Add a combined index/report command for migrated synthesis-text manifests
|
| 101 |
+
so the aggregate counts do not depend on ad hoc `jq` commands.
|
| 102 |
+
- [x] Add an explicit-field converter for remote toolkit JSONL/TSV/CSV parallel
|
| 103 |
+
corpora.
|
| 104 |
+
- [x] Add `audit-synthesis-text-promotion` so remote synthesis text must carry
|
| 105 |
+
split policy and must not overlap excluded eval/reference text before
|
| 106 |
+
rendering.
|
| 107 |
+
- [x] Add `split-synthesis-text` so promoted train/eval synthesis text is
|
| 108 |
+
grouped by `text_sha256`, stamped with split policy, and promotion-audited.
|
| 109 |
+
- [x] Smoke-run the converter on real remote Magar toolkit JSONL and pass
|
| 110 |
+
`audit-synthesis-text-manifest` with Magar/Nepali Devanagari gates.
|
| 111 |
+
- [x] Smoke-run that converter on selected Limbu, Rai, Dhimal, Gurung, and
|
| 112 |
+
Magar remote toolkit files, then audit the emitted synthesis-text manifests.
|
| 113 |
+
- [ ] Promote remote toolkit smoke commands into full split synthesis-text
|
| 114 |
+
manifests that pass `split-synthesis-text` with exclusion sets and license
|
| 115 |
+
policy appropriate to the run.
|
| 116 |
+
- [ ] Prioritize remaining converters in this order: Limbu Pangloss XML/TSV,
|
| 117 |
+
Bible Brain text manifests, Nepali prep TSV/transcript roots.
|
| 118 |
+
- [ ] Record license state per source. Bible Brain and derived text/audio roots
|
| 119 |
+
stay `pending_review` until cleared.
|
| 120 |
+
- [ ] Emit source path, source SHA-256, text SHA-256, language, script, Unicode
|
| 121 |
+
block counts, split policy, and `claim_evidence_eligible=false` for every
|
| 122 |
+
synthesis text row.
|
| 123 |
+
- [ ] Deduplicate synthesis text against all real validation references before
|
| 124 |
+
rendering.
|
| 125 |
+
- [ ] Render promoted train/eval synthesis text with verified fonts and
|
| 126 |
+
`--require-font` before using labels for PaddleOCR training.
|
| 127 |
+
- [ ] Verify fonts and Unicode renderability before rendering new scripts.
|
| 128 |
+
- [x] Add Namdhinggo 3.100 as the primary Limbu/Sirijonga font candidate and
|
| 129 |
+
keep Noto Sans Limbu as a fallback runtime font.
|
| 130 |
+
- [x] Add `configs/namdhinggo-font-assets-v1.json` so Namdhinggo 3.100 is
|
| 131 |
+
reproducibly provisioned from the upstream SIL release ZIP with package and
|
| 132 |
+
extracted-member SHA-256 checks.
|
| 133 |
+
- [x] Add `ocrtech inventory-fonts` for directory/ZIP font collection audits
|
| 134 |
+
with SHA-256, cmap readability, script coverage, duplicate detection, and
|
| 135 |
+
JSON/CSV/Markdown outputs.
|
| 136 |
+
- [x] Inventory current runtime font corpus:
|
| 137 |
+
`outputs/font-inventory-current-2026-06-16` has `17` font files, `7`
|
| 138 |
+
cmap-readable fonts, and `0` duplicate SHA groups.
|
| 139 |
+
- [x] Inventory `/Users/cdjk/github/llm/ltk/fonts.zip`: found `janaki.ttf` and
|
| 140 |
+
`madan.ttf`; both are cmap-readable but do not match registry Unicode ranges,
|
| 141 |
+
so they stay legacy/conversion leads rather than OCR rendering fonts.
|
| 142 |
+
- [x] Probed `cdjk@100.110.148.43` under `~/github/llm` for font assets and
|
| 143 |
+
copied direct font files into ignored `data/fonts/remote-cdjk-2026-06-16`.
|
| 144 |
+
- [x] Inventory remote-copied fonts:
|
| 145 |
+
`outputs/font-inventory-remote-cdjk-2026-06-16` has `4` readable fonts. Only
|
| 146 |
+
Noto Sans Devanagari covers registered Devanagari targets; the 8-bit
|
| 147 |
+
Devanagari font stays a legacy conversion lead.
|
| 148 |
+
- [x] Download and inventory Google Fonts OFL batch:
|
| 149 |
+
Kanchenjunga Regular/Bold for Kirat Rai, Noto Sans Tirhuta for Tirhuta,
|
| 150 |
+
Noto Serif Devanagari for print-style Devanagari, and Noto Sans Ol Chiki
|
| 151 |
+
Variable for Santhali style diversity.
|
| 152 |
+
- [x] Add `configs/nepal-font-assets-v1.json` plus
|
| 153 |
+
`ocrtech prepare-font-assets` so the Google Fonts batch is reproducibly
|
| 154 |
+
provisioned with SHA-256 checks instead of manual curl commands.
|
| 155 |
+
- [x] Add `ocrtech audit-font-asset-readiness` to require passing asset
|
| 156 |
+
preparation, font inventory, and all-script renderability evidence before
|
| 157 |
+
treating a runtime font corpus as synthesis-ready.
|
| 158 |
+
- [x] Connect rendered synthesis audits to font readiness evidence via
|
| 159 |
+
`--font-readiness-report`; rendered rows now fail training-bound audits when
|
| 160 |
+
their `font_sha256` is not in the passed readiness manifest.
|
| 161 |
+
- [x] Reaudit the Namdhinggo-rendered Limbu/Sirijonga train/eval split against
|
| 162 |
+
Namdhinggo font readiness evidence:
|
| 163 |
+
`train=150`, `eval=27`, `issues=0`, `warnings=0`.
|
| 164 |
+
- [x] Add deterministic rendered-line degradation profiles for PaddleOCR
|
| 165 |
+
recognizer training: `clean`, `scan`, `low_light`, `uneven_light`,
|
| 166 |
+
`glare_light`, `camera_left`, `camera_right`, `camera_top`,
|
| 167 |
+
`camera_bottom`, and `phone_photo`. Every generated image now records the
|
| 168 |
+
selected profile, seed, and parameters in manifest metadata.
|
| 169 |
+
- [x] Add `ocrtech audit-rendered-degradation-split` to verify expected
|
| 170 |
+
degradation profiles per text hash and reject train/eval `text_sha256`
|
| 171 |
+
overlap in augmented rendered bundles.
|
| 172 |
+
- [x] Generate and audit the full Namdhinggo augmented Limbu/Sirijonga rendered
|
| 173 |
+
split: `900` train rows, `162` eval rows, six profiles per text hash,
|
| 174 |
+
`issues=0`, `warnings=0`.
|
| 175 |
+
- [x] Materialize the Limbu/Sirijonga augmentation ablation spec:
|
| 176 |
+
`configs/limbu-sirijonga-augmentation-ablation-v1.json`, with clean,
|
| 177 |
+
lighting, and camera arms plus training recipe directories for each arm.
|
| 178 |
+
- [x] Audit all ablation arms for profile balance and train/eval text-hash
|
| 179 |
+
isolation: clean `150/27`, lighting `450/81`, camera `600/108`,
|
| 180 |
+
all `issues=0`, `warnings=0`.
|
| 181 |
+
- [x] Include illumination and capture-angle robustness in the first
|
| 182 |
+
Sirijonga augmentation plan: `low_light`, `uneven_light`, `glare_light`,
|
| 183 |
+
`camera_left`, `camera_right`, `camera_top`, `camera_bottom`, and
|
| 184 |
+
`phone_photo` are now deterministic profiles with recorded seeds and
|
| 185 |
+
parameters.
|
| 186 |
+
- [x] Upgrade camera/phone-photo profiles from affine shear to deterministic
|
| 187 |
+
perspective transforms, and require `source_quad`, `output_quad`, and
|
| 188 |
+
`perspective_coefficients` in rendered-manifest audits.
|
| 189 |
+
- [x] Tighten rendered-manifest audits so lighting profiles must record
|
| 190 |
+
numeric brightness/contrast, gradient/blend, or glare metadata instead of
|
| 191 |
+
relying on profile names alone.
|
| 192 |
+
- [x] Add `ocrtech audit-augmentation-ablation` so Limbu/Sirijonga synthetic
|
| 193 |
+
ablation arms cannot be treated as promotion-ready without a passed real
|
| 194 |
+
Gorkhapatra verified-reference review audit.
|
| 195 |
+
- [x] Run the current ablation promotion audit:
|
| 196 |
+
`passed=False arms=3 real_evidence=0 verified_references=0 issues=1`.
|
| 197 |
+
All synthetic arms pass internally, but the intended blocker is missing real
|
| 198 |
+
verified Gorkhapatra evidence.
|
| 199 |
+
- [ ] Run a controlled Limbu/Sirijonga augmentation ablation: clean-only vs
|
| 200 |
+
lighting-only vs camera-angle/phone-photo mix, evaluated on real
|
| 201 |
+
Gorkhapatra/Naya Nepal pages and held-out synthetic rows without text-hash
|
| 202 |
+
leakage.
|
| 203 |
+
- [x] Promote coverage-proven Google Fonts candidates into
|
| 204 |
+
`languages/nepal-v1.json` and rerun all-script renderability:
|
| 205 |
+
`passed=True scripts=22 candidates=58 rendered=0`.
|
| 206 |
+
- [ ] Collect and inventory additional print-style Unicode Devanagari, Ol
|
| 207 |
+
Chiki, Kirat Rai, Tibetan/Tamyig, and other Nepal-script fonts with license
|
| 208 |
+
notes before using them for synthesis diversity.
|
| 209 |
+
|
| 210 |
+
## Gorkhapatra Validation Path
|
| 211 |
+
|
| 212 |
+
- [ ] Start from `https://gorkhapatraonline.com/publications/naya-nepal`.
|
| 213 |
+
- [x] Support paginated Naya Nepal publication discovery with
|
| 214 |
+
`audit-gorkhapatra-language-pages --publication-pages N`; Limbu appears on
|
| 215 |
+
page 9 in the 2026-06-16 live scan.
|
| 216 |
+
- [x] Capture article publication dates and language labels.
|
| 217 |
+
- [x] Query the epaper endpoint for the matching publication date:
|
| 218 |
+
`https://epaper.gorkhapatraonline.com/epaper-list?...&slug=gorkhapatra`.
|
| 219 |
+
- [x] Download the direct PDF from the viewer `file=` path, not the PDF.js
|
| 220 |
+
shell.
|
| 221 |
+
- [x] Search/render the language page, including `bhasa pristha` /
|
| 222 |
+
`भाषा पृष्ठ` candidates.
|
| 223 |
+
- [x] Extract PDF-native text spans, font names, bboxes, codepoints, and
|
| 224 |
+
embedded fonts before OCR/manual transcription. This is now implemented as
|
| 225 |
+
`ocrtech extract-gorkhapatra-language-page-pdf-text`.
|
| 226 |
+
- [x] Run the PDF-native extractor on the current Limbu pack:
|
| 227 |
+
`passed=True samples=2/2 spans=776 target_spans=304 fonts=10 warnings=0`.
|
| 228 |
+
Page 6 (`gorkhapatra-language-page-0001-01`) has `304`
|
| 229 |
+
`NamdhinggoSILL` spans and extracted embedded font
|
| 230 |
+
`UFNWLJ+NamdhinggoSILL`; page 7 has `0` target-font spans.
|
| 231 |
+
- [x] Compare the blog image, PDF page, and PDF font metadata to choose the
|
| 232 |
+
exact page candidate. Current conclusion: page 6 is the Limbu/Sirijonga
|
| 233 |
+
page; page 7 is the same-date non-Limbu page.
|
| 234 |
+
- [ ] Build or locate a Namdhinggo/Gorkhapatra legacy encoding converter to
|
| 235 |
+
Unicode Limbu (`U+1900..U+194F`) so extracted PDF spans can become reference
|
| 236 |
+
text instead of manual transcription burden.
|
| 237 |
+
- [ ] Finalize only after manual review accepts exactly one candidate for the
|
| 238 |
+
publication/language/date group.
|
| 239 |
+
- [x] Add a candidate-language filter to the Gorkhapatra language-page review
|
| 240 |
+
audit/finalize commands so Limbu can be gated independently from other
|
| 241 |
+
accepted Naya Nepal languages on the same PDF.
|
| 242 |
+
- [x] Run the filtered Limbu verified-reference audit:
|
| 243 |
+
`passed=False samples=2 groups=1 accepted=1 rejected=1 issues=1 warnings=0`.
|
| 244 |
+
The only blocker is empty verified reference text for
|
| 245 |
+
`gorkhapatra-language-page-0001-01`.
|
| 246 |
+
- [x] Add a transcription work-order generator for accepted Gorkhapatra
|
| 247 |
+
language-page rows so manual labeling is explicit, auditable, and scoped by
|
| 248 |
+
target language.
|
| 249 |
+
- [x] Generate the Limbu work order:
|
| 250 |
+
`passed=True samples=1 blocked=1 verified=0 warnings=0`. The selected work
|
| 251 |
+
item is `gorkhapatra-language-page-0001-01`, page 6 of the 2083-02-12
|
| 252 |
+
Gorkhapatra PDF.
|
| 253 |
+
- [x] Correct the work-order priority: for Gorkhapatra epapers, PDF-native
|
| 254 |
+
extraction is upstream of OCR and manual transcription. The transcription
|
| 255 |
+
dashboard is fallback/verification only, especially for Limbu/Sirijonga where
|
| 256 |
+
OCR candidates are currently invalid.
|
| 257 |
+
- [x] Add `prepare-gorkhapatra-language-page-assisted-references` for
|
| 258 |
+
non-claim-ready machine OCR draft references. Drafts preserve
|
| 259 |
+
`reference_status=draft`, `claim_evidence_eligible=false`,
|
| 260 |
+
`machine_generated=true`, and `requires_human_review=true`.
|
| 261 |
+
- [x] Add `prepare-gorkhapatra-language-page-ocr-sidecars` so accepted
|
| 262 |
+
Gorkhapatra rows can produce reproducible OCR sidecars before assisted
|
| 263 |
+
reference drafting.
|
| 264 |
+
- [x] Run PaddleOCR sidecar generation for the current Limbu page on k2:
|
| 265 |
+
`pages=1`, `lines=299`, `nonzero_bbox=299`,
|
| 266 |
+
`average_confidence=0.5347613170992171`.
|
| 267 |
+
- [x] Rerun the assisted-reference command on the current Limbu page:
|
| 268 |
+
`passed=True samples=1 assisted=1 failed=0 skipped=3 warnings=0`. The draft
|
| 269 |
+
now contains `298` non-empty OCR text lines and remains non-claim-ready.
|
| 270 |
+
- [x] Add `prepare-gorkhapatra-language-page-verification-bundle` for
|
| 271 |
+
line-level human correction sheets from assisted references.
|
| 272 |
+
- [x] Add `apply-gorkhapatra-language-page-verification-bundle` to convert a
|
| 273 |
+
completed line-review CSV into verified `.ref.json` files and an updated
|
| 274 |
+
page-review CSV, while blocking `pending`, `needs_context`, unknown, or empty
|
| 275 |
+
reviewed rows.
|
| 276 |
+
- [x] Tighten verified-reference application so claim-eligible references also
|
| 277 |
+
require explicit table, figure, and caption review declarations. Checked
|
| 278 |
+
absent is acceptable; silently inheriting empty draft arrays is not.
|
| 279 |
+
- [x] Generate the current Limbu verification bundle:
|
| 280 |
+
`passed=True samples=1 lines=298 crops=298 missing_bboxes=0 warnings=0`.
|
| 281 |
+
The CSV review sheet and line crops are ready for human correction.
|
| 282 |
+
- [x] Add an interactive HTML review dashboard to every verification bundle so
|
| 283 |
+
reviewers can scan line crops, ids, order, confidence, and OCR text, set
|
| 284 |
+
`review_status`, enter corrected text, add notes, and export a reviewed CSV
|
| 285 |
+
that still must pass the audit/apply gates.
|
| 286 |
+
- [x] Add `audit-gorkhapatra-language-page-verification-csv` for dry-run
|
| 287 |
+
status counts and blocker reports before writing verified references.
|
| 288 |
+
- [x] Add `split-gorkhapatra-language-page-verification-csv` to produce
|
| 289 |
+
reviewer-sized CSV/HTML batches while preserving line ids and order.
|
| 290 |
+
- [x] Add `merge-gorkhapatra-language-page-verification-batches` so reviewed
|
| 291 |
+
batch CSVs can be recombined only when every original `(sample_id, line_id)`
|
| 292 |
+
appears exactly once.
|
| 293 |
+
- [x] Add `assign-gorkhapatra-language-page-verification-batches` so split
|
| 294 |
+
batches have an explicit reviewer/due-date ledger for paper-grade provenance.
|
| 295 |
+
- [x] Run the real Limbu split/merge smoke:
|
| 296 |
+
`split passed=True samples=1 lines=298 batches=6 batch_size=50 warnings=0`;
|
| 297 |
+
untouched-batch merge `passed=True source_lines=298 merged_lines=298 batch_files=6 issues=0 warnings=0`;
|
| 298 |
+
merged audit still blocks with `pending=298`.
|
| 299 |
+
- [x] Run the current Limbu verification CSV audit:
|
| 300 |
+
`passed=False samples=1 lines=298 ready=0 dropped=0 blocked=298 issues=298 warnings=0`;
|
| 301 |
+
all rows remain `pending`.
|
| 302 |
+
- [ ] Write structured references with text, reading order, tables, figures,
|
| 303 |
+
captions, and Markdown where present.
|
| 304 |
+
- [ ] Run `audit-gorkhapatra-language-page-review` and `audit-references`
|
| 305 |
+
before using the pack in claim-grade validation.
|
| 306 |
+
|
| 307 |
+
## Paper Evidence Checklist
|
| 308 |
+
|
| 309 |
+
- [ ] Every benchmark report records input hashes, output artifact hashes,
|
| 310 |
+
model-card hashes, latency, memory, and fallback usage.
|
| 311 |
+
- [ ] Every public or real eval row has a parsed and audited reference.
|
| 312 |
+
- [ ] Claim review rejects stale artifacts, missing references, malformed
|
| 313 |
+
structured references, and synthetic-only evidence.
|
| 314 |
+
- [ ] Error analysis includes failures by language, script, layout type,
|
| 315 |
+
degradation, table presence, and fallback trigger reason.
|
| 316 |
+
- [ ] Paper tables separate OCR metrics, document-structure metrics, resource
|
| 317 |
+
metrics, and fallback metrics.
|
| 318 |
+
|
| 319 |
+
## Limbu/Sirijonga Current Evidence
|
| 320 |
+
|
| 321 |
+
- [x] Built split-safe Limbu Unicode synthesis text:
|
| 322 |
+
`outputs/sirijonga-limbu-split-2026-06-16/train.jsonl` and
|
| 323 |
+
`outputs/sirijonga-limbu-split-2026-06-16/eval.jsonl`.
|
| 324 |
+
- [x] Split policy is
|
| 325 |
+
`split_synthesis_text:text_sha256:seed=17:eval_ratio=0.15`, with `170`
|
| 326 |
+
train rows, `30` eval rows, and `200` text-hash groups.
|
| 327 |
+
- [x] Rendered the pure Limbu/Sirijonga subset with font provenance:
|
| 328 |
+
`150` train lines and `27` eval lines passed rendered-line audits using
|
| 329 |
+
Noto Sans Limbu.
|
| 330 |
+
- [x] Rerendered the same split with Namdhinggo Regular; this is now the
|
| 331 |
+
preferred Sirijonga training bundle.
|
| 332 |
+
- [x] Reaudited the Namdhinggo-rendered split with
|
| 333 |
+
`--font-readiness-report outputs/font-asset-readiness-namdhinggo-2026-06-16/font-asset-readiness.json`;
|
| 334 |
+
the training bundle is now tied to reproducible Namdhinggo asset provenance.
|
| 335 |
+
- [x] Regenerated the preferred Namdhinggo split with explicit degradation
|
| 336 |
+
profiles after confirming the clean split remained reproducible. The
|
| 337 |
+
degradation split audit confirms clean/degraded variants stay on the same
|
| 338 |
+
split side.
|
| 339 |
+
- [x] Add a Limbu/Sirijonga PaddleOCR dictionary covering `U+1900..U+194F`
|
| 340 |
+
plus accepted punctuation.
|
| 341 |
+
- [x] Audit that dictionary against the Namdhinggo-rendered train/eval label
|
| 342 |
+
files: `76` dictionary characters, `68` required Limbu-range characters,
|
| 343 |
+
`64` observed label characters, `0` missing required, `0` missing labels.
|
| 344 |
+
Label spaces are covered by PaddleOCR `use_space_char=true`, not by a literal
|
| 345 |
+
whitespace dictionary row.
|
| 346 |
+
- [ ] Find or label real Gorkhapatra/Naya Nepal Sirijonga pages before any
|
| 347 |
+
claim-grade evaluation.
|
| 348 |
+
- [x] Found the real Limbu Naya Nepal item:
|
| 349 |
+
`https://gorkhapatraonline.com/news/207555`, date `2083-02-12`, article image
|
| 350 |
+
`https://gorkhapatraonline.com/storage/media/360954/conversions/6-original.jpg`.
|
| 351 |
+
- [x] Matched that date to Gorkhapatra epaper PDF
|
| 352 |
+
`2026-05-26-03-43-57-2083-02-12.pdf`, SHA-256
|
| 353 |
+
`afe1d3d2e64248745d94d511e5ca456abddd3ad828480baff87263029b987cd8`.
|
| 354 |
+
- [x] Rendered same-date page candidates and visually disambiguated:
|
| 355 |
+
page 6 is Limbu/Sirijonga, page 7 is Majhi/Devanagari.
|
| 356 |
+
- [x] Created an accepted-row reviewer bundle for manual reference labeling:
|
| 357 |
+
`data/limbu-gorkhapatra-language-page-reviewer-bundle-2026-06-16`, with
|
| 358 |
+
`2` images and `2` draft references.
|
| 359 |
+
- [x] Created a Limbu transcription work order:
|
| 360 |
+
`data/limbu-gorkhapatra-language-page-transcription-work-order-2026-06-16`.
|
| 361 |
+
It confirms one accepted Limbu page is blocked only by the empty reference
|
| 362 |
+
JSON, not by missing image/PDF provenance.
|
| 363 |
+
- [x] Created a Limbu assisted-reference draft run:
|
| 364 |
+
`data/limbu-gorkhapatra-language-page-assisted-references-2026-06-16`.
|
| 365 |
+
It now contains PaddleOCR-assisted text suggestions from k2 and keeps the
|
| 366 |
+
draft non-claim-ready.
|
| 367 |
+
- [x] Created a Limbu line-level verification bundle:
|
| 368 |
+
`data/limbu-gorkhapatra-language-page-verification-bundle-2026-06-16`, with
|
| 369 |
+
`298` line crops, a pending review CSV, and an HTML review contact sheet.
|
| 370 |
+
- [x] Confirmed the correct PDF-native browser review surface:
|
| 371 |
+
`file:///Users/cdjk/github/llm/ocr-tech/data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html`.
|
| 372 |
+
Use this full-page clickable PDF-span dashboard for verification. Do not use
|
| 373 |
+
the PDF-native verification-bundle HTML as the review UI; that was identified
|
| 374 |
+
as the wrong/broken interface.
|
| 375 |
+
- [ ] Fill the verification CSV with explicit `accept`, `corrected`, or `drop`
|
| 376 |
+
line statuses, either directly or via
|
| 377 |
+
`split-gorkhapatra-language-page-verification-csv` batches. If batches are
|
| 378 |
+
used, merge them with `merge-gorkhapatra-language-page-verification-batches`.
|
| 379 |
+
Then run `audit-gorkhapatra-language-page-verification-csv`, then run
|
| 380 |
+
`apply-gorkhapatra-language-page-verification-bundle` to produce verified
|
| 381 |
+
references and an updated review CSV with `--tables-reviewed`,
|
| 382 |
+
`--figures-reviewed`, and `--captions-reviewed` after those structures are checked.
|
| 383 |
+
- [ ] Fill verified text/layout references for the accepted Limbu page; strict
|
| 384 |
+
review audit currently fails as intended until that reference is complete.
|
| 385 |
+
- [ ] Re-run the Limbu-only gate after transcription:
|
| 386 |
+
`audit-gorkhapatra-language-page-review ... --candidate-language लिम्बू --require-verified-references`.
|
ocr-tech-mps-integration/docs/CDJK_LANGUAGE_CORPUS_MIGRATION.md
ADDED
|
@@ -0,0 +1,310 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# cdjk Language Corpus Migration
|
| 2 |
+
|
| 3 |
+
Run date: 2026-06-16.
|
| 4 |
+
|
| 5 |
+
This document records the first local migration of language corpora and
|
| 6 |
+
dictionaries from `cdjk@100.110.148.43` for multilingual OCR synthesis. The raw
|
| 7 |
+
copy lives under ignored `data/`; the reproducible policy is tracked in
|
| 8 |
+
`configs/cdjk-language-corpus-migration-v1.json`.
|
| 9 |
+
|
| 10 |
+
## Source Host
|
| 11 |
+
|
| 12 |
+
```text
|
| 13 |
+
host: cdjk@100.110.148.43
|
| 14 |
+
machine: cdjks-MacBook-Pro.local
|
| 15 |
+
home: /Users/cdjk
|
| 16 |
+
external volume: /Volumes/TRANSCEND
|
| 17 |
+
external volume size: 3.6Ti
|
| 18 |
+
external volume free at inventory: 2.0Ti
|
| 19 |
+
```
|
| 20 |
+
|
| 21 |
+
Local free space at copy time was about `33GiB`, so the migration was
|
| 22 |
+
selective. We copied text-like corpus resources and skipped audio, video,
|
| 23 |
+
models, checkpoints, PDFs, images, git metadata, caches, and virtualenvs.
|
| 24 |
+
|
| 25 |
+
## Copied Roots
|
| 26 |
+
|
| 27 |
+
Destination:
|
| 28 |
+
|
| 29 |
+
```text
|
| 30 |
+
data/external-language-resources/cdjk-100.110.148.43/
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
Copied roots:
|
| 34 |
+
|
| 35 |
+
```text
|
| 36 |
+
dhimal-toolkit
|
| 37 |
+
gurung-language-toolkit
|
| 38 |
+
limbu-speech-toolkit
|
| 39 |
+
ltk
|
| 40 |
+
magar-toolkit
|
| 41 |
+
rai-speech-toolkit
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
Copy result:
|
| 45 |
+
|
| 46 |
+
```text
|
| 47 |
+
files transferred: 2307
|
| 48 |
+
files after copy: 2308
|
| 49 |
+
local size: 1.6G
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
Suffix counts:
|
| 53 |
+
|
| 54 |
+
```text
|
| 55 |
+
json 1071
|
| 56 |
+
txt 686
|
| 57 |
+
md 209
|
| 58 |
+
jsonl 179
|
| 59 |
+
tsv 155
|
| 60 |
+
csv 4
|
| 61 |
+
yml 3
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
+
Per-root local size:
|
| 65 |
+
|
| 66 |
+
```text
|
| 67 |
+
gurung-language-toolkit 41M
|
| 68 |
+
magar-toolkit 43M
|
| 69 |
+
ltk 67M
|
| 70 |
+
dhimal-toolkit 94M
|
| 71 |
+
limbu-speech-toolkit 555M
|
| 72 |
+
rai-speech-toolkit 841M
|
| 73 |
+
total 1.6G
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
## Resource Audit
|
| 77 |
+
|
| 78 |
+
Command:
|
| 79 |
+
|
| 80 |
+
```bash
|
| 81 |
+
uv run ocrtech audit-synthesis-resources \
|
| 82 |
+
--root data/external-language-resources/cdjk-100.110.148.43/ltk --label ltk \
|
| 83 |
+
--root data/external-language-resources/cdjk-100.110.148.43/limbu-speech-toolkit --label limbu-speech-toolkit \
|
| 84 |
+
--root data/external-language-resources/cdjk-100.110.148.43/rai-speech-toolkit --label rai-speech-toolkit \
|
| 85 |
+
--root data/external-language-resources/cdjk-100.110.148.43/gurung-language-toolkit --label gurung-language-toolkit \
|
| 86 |
+
--root data/external-language-resources/cdjk-100.110.148.43/magar-toolkit --label magar-toolkit \
|
| 87 |
+
--root data/external-language-resources/cdjk-100.110.148.43/dhimal-toolkit --label dhimal-toolkit \
|
| 88 |
+
--out outputs/cdjk-language-resource-audit-2026-06-16 \
|
| 89 |
+
--max-text-samples-per-root 8
|
| 90 |
+
```
|
| 91 |
+
|
| 92 |
+
Result:
|
| 93 |
+
|
| 94 |
+
```text
|
| 95 |
+
passed=True roots=6/6 files=2308 warnings=0
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
## Prepared Text Manifests
|
| 99 |
+
|
| 100 |
+
Limbu Limdic:
|
| 101 |
+
|
| 102 |
+
```bash
|
| 103 |
+
uv run ocrtech prepare-limbu-limdic-text \
|
| 104 |
+
--source data/external-language-resources/cdjk-100.110.148.43/ltk/limbu/data/limdic_dictionary.json \
|
| 105 |
+
--source data/external-language-resources/cdjk-100.110.148.43/ltk/limbu/data/limdic_dictionary.tsv \
|
| 106 |
+
--out data/synthesis-text/cdjk-limbu-limdic-2026-06-16 \
|
| 107 |
+
--dataset cdjk-limbu-limdic
|
| 108 |
+
```
|
| 109 |
+
|
| 110 |
+
Result:
|
| 111 |
+
|
| 112 |
+
```text
|
| 113 |
+
passed=True samples=26152 duplicates=26628 warnings=0
|
| 114 |
+
script_counts:
|
| 115 |
+
Limbu: 8772
|
| 116 |
+
Devanagari: 8747
|
| 117 |
+
Mixed_Devanagari_Latin: 8616
|
| 118 |
+
Mixed_Limbu_Devanagari: 14
|
| 119 |
+
Mixed_Limbu_Latin: 2
|
| 120 |
+
Latin: 1
|
| 121 |
+
```
|
| 122 |
+
|
| 123 |
+
The Limbu-only subset was created for Namdhinggo rendering:
|
| 124 |
+
|
| 125 |
+
```bash
|
| 126 |
+
mkdir -p data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered
|
| 127 |
+
jq -c 'select(.script == "Limbu")' \
|
| 128 |
+
data/synthesis-text/cdjk-limbu-limdic-2026-06-16/limbu-limdic-text.jsonl \
|
| 129 |
+
> data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered/limbu-only.jsonl
|
| 130 |
+
|
| 131 |
+
uv run ocrtech audit-synthesis-text-manifest \
|
| 132 |
+
data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered/limbu-only.jsonl \
|
| 133 |
+
--out outputs/cdjk-limbu-only-text-audit-2026-06-16 \
|
| 134 |
+
--require-language limbu \
|
| 135 |
+
--require-script Limbu \
|
| 136 |
+
--min-samples 100
|
| 137 |
+
```
|
| 138 |
+
|
| 139 |
+
Result:
|
| 140 |
+
|
| 141 |
+
```text
|
| 142 |
+
passed=True samples=8772 claim_eligible=0 issues=0 warnings=0
|
| 143 |
+
```
|
| 144 |
+
|
| 145 |
+
Tamang:
|
| 146 |
+
|
| 147 |
+
```bash
|
| 148 |
+
uv run ocrtech prepare-tamang-text \
|
| 149 |
+
--source data/external-language-resources/cdjk-100.110.148.43/ltk/tamang/data/neptam_train.csv \
|
| 150 |
+
--source data/external-language-resources/cdjk-100.110.148.43/ltk/tamang/data/neptam_test.csv \
|
| 151 |
+
--source data/external-language-resources/cdjk-100.110.148.43/ltk/tamang/data/western_tamang_dictionary.tsv \
|
| 152 |
+
--out data/synthesis-text/cdjk-tamang-2026-06-16 \
|
| 153 |
+
--dataset cdjk-tamang-real-text
|
| 154 |
+
```
|
| 155 |
+
|
| 156 |
+
Result:
|
| 157 |
+
|
| 158 |
+
```text
|
| 159 |
+
passed=True samples=46314 duplicates=892 warnings=0
|
| 160 |
+
audit: passed=True samples=46314 claim_eligible=0 issues=0 warnings=0
|
| 161 |
+
```
|
| 162 |
+
|
| 163 |
+
Magar:
|
| 164 |
+
|
| 165 |
+
```bash
|
| 166 |
+
uv run ocrtech prepare-magar-text \
|
| 167 |
+
--source data/external-language-resources/cdjk-100.110.148.43/ltk/magar/data/western_magar_dictionary.tsv \
|
| 168 |
+
--source data/external-language-resources/cdjk-100.110.148.43/ltk/magar/data/western_magar_dictionary.json \
|
| 169 |
+
--out data/synthesis-text/cdjk-magar-2026-06-16 \
|
| 170 |
+
--dataset cdjk-magar-western-dictionary
|
| 171 |
+
```
|
| 172 |
+
|
| 173 |
+
Result:
|
| 174 |
+
|
| 175 |
+
```text
|
| 176 |
+
passed=True samples=1182 duplicates=1246 warnings=0
|
| 177 |
+
audit: passed=True samples=1182 claim_eligible=0 issues=0 warnings=0
|
| 178 |
+
```
|
| 179 |
+
|
| 180 |
+
Important schema note: `magar-toolkit/data/lexicon/mgp-npi.tsv` and
|
| 181 |
+
`magar-toolkit/data/mono/magar-dhut-clean.tsv` are useful, but they do not
|
| 182 |
+
match the Western Magar dictionary converter schema. They need an explicit
|
| 183 |
+
field-mapped converter path before use.
|
| 184 |
+
|
| 185 |
+
## Field-Mapped Toolkit Sweep
|
| 186 |
+
|
| 187 |
+
After the first Limbu/Tamang/Magar pass, the explicit-field converter was run
|
| 188 |
+
on Gurung, Dhimal, extra Magar, and Rai dialect sources. Mixed schemas were
|
| 189 |
+
split instead of being forced into one command. Empty split files were skipped.
|
| 190 |
+
|
| 191 |
+
Results:
|
| 192 |
+
|
| 193 |
+
| Manifest | Samples | Duplicates | Notes |
|
| 194 |
+
| --- | ---: | ---: | --- |
|
| 195 |
+
| Gurung toolkit | 17,061 | 79 | `gvr` target text plus Nepali source text |
|
| 196 |
+
| Dhimal parallel | 37,504 | 11,936 | Bible/parallel rows |
|
| 197 |
+
| Dhimal mono | 740 | 2 | Epustakalaya monolingual rows |
|
| 198 |
+
| Magar parallel extra | 20,127 | 1,391 | `mgp/npi` lexicon and MT rows |
|
| 199 |
+
| Magar mono extra | 3,441 | 0 | `mgp/source/mscore` monolingual rows |
|
| 200 |
+
| Rai Bantawa | 747 | 0 | mostly Latin phonetic text |
|
| 201 |
+
| Rai Chamling | 23 | 0 | Devanagari rows |
|
| 202 |
+
| Rai Kulung | 7,956 | 1 | Devanagari rows |
|
| 203 |
+
| Rai Athpariya | 7,861 | 5 | Devanagari rows |
|
| 204 |
+
| Rai Khaling | 1,166 | 84 | mixed Devanagari/Latin source pool |
|
| 205 |
+
| Rai Yakkha | 1,357 | 1,438 | Devanagari rows |
|
| 206 |
+
| Rai Dumi | 1,453 | 42 | Devanagari rows |
|
| 207 |
+
| Rai Thulung | 2,825 | 0 | mostly Latin phonetic text |
|
| 208 |
+
| Rai Koyu | 819 | 0 | mostly Latin phonetic text |
|
| 209 |
+
| Rai Lohorung | 10,202 | 42 | Devanagari rows |
|
| 210 |
+
| Rai Sunwar | 32,165 | 1,445 | Devanagari rows |
|
| 211 |
+
| Rai Limbu | 9,117 | 27 | Devanagari-transcribed Limbu plus some Latin |
|
| 212 |
+
|
| 213 |
+
Aggregate generated synthesis-text inventory after this sweep:
|
| 214 |
+
|
| 215 |
+
```text
|
| 216 |
+
manifests: 20
|
| 217 |
+
samples: 228212
|
| 218 |
+
duplicates removed/reported by converters: 45258
|
| 219 |
+
script_counts:
|
| 220 |
+
Devanagari: 202560
|
| 221 |
+
Mixed_Devanagari_Latin: 10337
|
| 222 |
+
Latin: 6521
|
| 223 |
+
Limbu: 8772
|
| 224 |
+
Mixed_Limbu_Devanagari: 14
|
| 225 |
+
Mixed_Limbu_Latin: 3
|
| 226 |
+
Other: 5
|
| 227 |
+
```
|
| 228 |
+
|
| 229 |
+
Interpretation:
|
| 230 |
+
|
| 231 |
+
- Limbu/Sirijonga has true Unicode Limbu rows from Limdic and can render with
|
| 232 |
+
Namdhinggo after filtering `script == Limbu`.
|
| 233 |
+
- Most Gurung, Dhimal, Magar, and Rai rows are Devanagari-transcribed language
|
| 234 |
+
text; these should render with a Devanagari font unless/until native-script
|
| 235 |
+
sources are found.
|
| 236 |
+
- Bantawa, Khaling, Koyu, and Thulung include substantial Latin phonetic data.
|
| 237 |
+
Those rows are useful for corpus/dictionary modeling, but they are not native
|
| 238 |
+
script OCR targets unless the research question explicitly includes romanized
|
| 239 |
+
field notation.
|
| 240 |
+
|
| 241 |
+
## Limbu Synthetic Smoke
|
| 242 |
+
|
| 243 |
+
First attempt rendered mixed-script rows with Namdhinggo and exposed tofu boxes
|
| 244 |
+
for Devanagari/Latin rows. That output is not training-valid. The corrected
|
| 245 |
+
smoke render uses only `script == Limbu`.
|
| 246 |
+
|
| 247 |
+
Correct command:
|
| 248 |
+
|
| 249 |
+
```bash
|
| 250 |
+
uv run ocrtech render-synthesis-text-lines \
|
| 251 |
+
data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered/limbu-only.jsonl \
|
| 252 |
+
--out data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only \
|
| 253 |
+
--font-path data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-Regular.ttf \
|
| 254 |
+
--limit 120 \
|
| 255 |
+
--degradation-profile clean \
|
| 256 |
+
--degradation-profile scan \
|
| 257 |
+
--degradation-profile low_light \
|
| 258 |
+
--degradation-profile uneven_light \
|
| 259 |
+
--degradation-profile camera_left \
|
| 260 |
+
--degradation-profile phone_photo
|
| 261 |
+
|
| 262 |
+
uv run ocrtech audit-rendered-synthesis-lines \
|
| 263 |
+
data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only/manifests/rendered-line-manifest.jsonl \
|
| 264 |
+
--label-file data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only/labels/train.txt \
|
| 265 |
+
--out outputs/cdjk-limbu-rendered-smoke-audit-2026-06-16-limbu-only
|
| 266 |
+
```
|
| 267 |
+
|
| 268 |
+
Result:
|
| 269 |
+
|
| 270 |
+
```text
|
| 271 |
+
passed=True samples=120 skipped=0 warnings=0
|
| 272 |
+
audit: passed=True samples=120 claim_eligible=0 issues=0 warnings=0
|
| 273 |
+
degradation_counts:
|
| 274 |
+
camera_left: 20
|
| 275 |
+
clean: 20
|
| 276 |
+
low_light: 20
|
| 277 |
+
phone_photo: 20
|
| 278 |
+
scan: 20
|
| 279 |
+
uneven_light: 20
|
| 280 |
+
```
|
| 281 |
+
|
| 282 |
+
Visual spot check confirmed actual Limbu glyphs in the corrected render.
|
| 283 |
+
|
| 284 |
+
## Repeatable Pattern for Each Language
|
| 285 |
+
|
| 286 |
+
1. Inventory copied source root with `audit-synthesis-resources`.
|
| 287 |
+
2. Verify actual file schemas before conversion.
|
| 288 |
+
3. Use a schema-specific converter or explicit field mapping.
|
| 289 |
+
4. Run `audit-synthesis-text-manifest`.
|
| 290 |
+
5. Filter by script before selecting a script-specific font.
|
| 291 |
+
6. Verify font renderability and font readiness.
|
| 292 |
+
7. Render train/eval with deterministic degradations.
|
| 293 |
+
8. Run `audit-rendered-synthesis-lines`.
|
| 294 |
+
9. Keep all synthetic rows marked `claim_evidence_eligible=false`.
|
| 295 |
+
10. Evaluate claims only on real held-out sources such as verified Gorkhapatra
|
| 296 |
+
pages.
|
| 297 |
+
|
| 298 |
+
## Next Checklist
|
| 299 |
+
|
| 300 |
+
- [ ] Add explicit field-mapped conversion recipes for Gurung, Dhimal, and Rai
|
| 301 |
+
dialect corpora.
|
| 302 |
+
- [ ] Add Magar converter support for `mgp/source/mscore` monolingual rows and
|
| 303 |
+
`mgp/npi` lexicon rows.
|
| 304 |
+
- [ ] Build a script/font matrix for each language before rendering.
|
| 305 |
+
- [ ] Create train/eval splits with `split-synthesis-text` before large-scale
|
| 306 |
+
rendering.
|
| 307 |
+
- [ ] Run large synthetic rendering on a machine with enough free disk, not on
|
| 308 |
+
this local 33GiB-limited workspace.
|
| 309 |
+
- [ ] Link rendered synthetic bundles to PaddleOCR recognizer recipes only
|
| 310 |
+
after font readiness and rendered audits pass.
|
ocr-tech-mps-integration/docs/CLAIM_REVIEW.md
ADDED
|
@@ -0,0 +1,333 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Claim Review
|
| 2 |
+
|
| 3 |
+
`review-claim` is the portfolio-level gate for a SOTA claim. `validate-claim` can tell you whether one experiment passed its configured gates. It cannot tell you whether you have enough evidence across the internal Nepali/English pack and public suites to say "better than what is out there." That is what `review-claim` is for.
|
| 4 |
+
|
| 5 |
+
## What It Audits
|
| 6 |
+
|
| 7 |
+
`review-claim` checks that:
|
| 8 |
+
|
| 9 |
+
- every required experiment slot is backed by a real `experiment.json`
|
| 10 |
+
- every referenced eval manifest and benchmark report still exists at the
|
| 11 |
+
recorded path, and the eval manifest matches the experiment's recorded
|
| 12 |
+
SHA-256
|
| 13 |
+
- successful benchmark rows are bound back to sample ids and input paths in the
|
| 14 |
+
reviewed eval manifest, and row slice labels must match the manifest entry
|
| 15 |
+
- the expected datasets or manifest patterns are present
|
| 16 |
+
- required baselines were actually run
|
| 17 |
+
- required metric families are present on successful rows for the required baselines
|
| 18 |
+
- required metric families are present on the slices they are supposed to
|
| 19 |
+
substantiate when `required_metric_slices` is configured
|
| 20 |
+
- benchmark reports do not contain error rows when the slot requires clean runs;
|
| 21 |
+
status counts are derived from `benchmark/report.json`, not trusted from the
|
| 22 |
+
copied `experiment.json` summary
|
| 23 |
+
- required validation status is present for internal claim-grade experiments,
|
| 24 |
+
and the referenced `validation.json` agrees with `experiment.json`
|
| 25 |
+
- required validation gates passed for the baseline/metric/slice comparisons
|
| 26 |
+
declared by `required_validation_gates`
|
| 27 |
+
- when configured, candidate fallback diagnostics are present and the aggregate
|
| 28 |
+
fallback trigger/failure rates stay under the slot limits
|
| 29 |
+
- required slices exist for the internal Nepali/English pack
|
| 30 |
+
- candidate experiments all point to the same candidate model config
|
| 31 |
+
- the candidate model card matches experiment hashes and passes `audit-model`,
|
| 32 |
+
with or without an explicit `--model-config`
|
| 33 |
+
- if candidate experiments record a fallback model card, that card exists,
|
| 34 |
+
matches the experiment SHA-256, and passes `audit-model`
|
| 35 |
+
- when configured, the candidate model card itself records the same fallback
|
| 36 |
+
component in `provenance.fallback_model_card`
|
| 37 |
+
- when configured, the candidate model card proves it includes a trained local
|
| 38 |
+
recognizer with artifacts, train/eval manifest hashes, and training-summary
|
| 39 |
+
provenance
|
| 40 |
+
- when configured, the candidate model card references a passing admission
|
| 41 |
+
validation report before it can enter portfolio review
|
| 42 |
+
- routed candidates such as `quality_select_composite`, `quality_ranked_ensemble`, and `script_select_composite` were calibrated on a separate eval manifest, not on one of the reviewed experiment manifests
|
| 43 |
+
- routed candidates also fail review if their selected calibration trial collapses to one engine on every calibration sample
|
| 44 |
+
- routed candidates also fail review if a reviewed benchmark shows they are exactly indistinguishable from one of their component baselines on paired holdout CER
|
| 45 |
+
|
| 46 |
+
Metric coverage is a hard gate because the target claim is a document-understanding
|
| 47 |
+
claim, not a plain OCR claim. A required experiment can now declare
|
| 48 |
+
`required_metrics`; each metric must appear as a finite value on at least one
|
| 49 |
+
successful row for every required baseline in that slot. For the internal
|
| 50 |
+
Nepali/English document pack, the SOTA config requires text OCR metrics
|
| 51 |
+
(`cer`, `wer`), document reconstruction
|
| 52 |
+
(`document_reconstruction_similarity`), table reconstruction
|
| 53 |
+
(`table_cell_f1`), reading order (`reading_order_pair_accuracy`), and figure/caption metrics
|
| 54 |
+
(`figure_detection_f1`, `figure_caption_cer`). Public table/document slots
|
| 55 |
+
declare their own metric set.
|
| 56 |
+
|
| 57 |
+
For claim-grade slots, also declare `required_metric_slices`. This prevents a
|
| 58 |
+
metric from satisfying the portfolio check on the wrong document type. For
|
| 59 |
+
example, `table_cell_f1` must be finite on a `table` or document-parsing row,
|
| 60 |
+
and `figure_caption_cer` must be finite on a `figure` row.
|
| 61 |
+
|
| 62 |
+
Metric presence is not enough. When a required slot includes `candidate` and
|
| 63 |
+
one or more external baselines, `review-claim` also requires paired metric
|
| 64 |
+
coverage on the same sample ids. If `required_metric_slices` is configured, the
|
| 65 |
+
pairing is checked on those slices; otherwise it is checked across all rows.
|
| 66 |
+
This prevents a SOTA review from passing because candidate metrics and baseline
|
| 67 |
+
metrics both exist somewhere in the report but never compare the same documents.
|
| 68 |
+
|
| 69 |
+
Sample coverage is also a hard gate. Every `require_baselines` entry must have a
|
| 70 |
+
successful benchmark row for every sample in the reviewed eval manifest. This
|
| 71 |
+
prevents a report from pointing at a hard eval pack while only running an easy
|
| 72 |
+
subset.
|
| 73 |
+
|
| 74 |
+
When a slot declares `require_validation_status`, `review-claim` reads the
|
| 75 |
+
referenced validation report and verifies `claim_status`. For `validated`, it
|
| 76 |
+
also requires `passed: true`. The copied status inside `experiment.json` is not
|
| 77 |
+
enough on its own. The validation artifact must also carry provenance linking it
|
| 78 |
+
to the same benchmark report, eval manifest, validation config, model cards,
|
| 79 |
+
and train manifest hashes referenced by the experiment.
|
| 80 |
+
|
| 81 |
+
Use `required_validation_gates` for claim-grade slots. Metric coverage proves
|
| 82 |
+
that a benchmark emitted a metric; passing validation gates prove the candidate
|
| 83 |
+
beat a named baseline under the configured sample count, slices, margins, and
|
| 84 |
+
statistical checks. Review requires each matched gate to carry quantitative
|
| 85 |
+
paired evidence, not just `passed: true`: positive `pairs`, finite candidate
|
| 86 |
+
and baseline means, finite absolute improvement, and win/loss/tie counts that
|
| 87 |
+
sum to the pair count. It also re-checks the threshold fields embedded in the
|
| 88 |
+
gate result, including `min_pairs`, margins, `min_pair_win_rate`, `max_p_value`,
|
| 89 |
+
and `require_ci_above_zero`, so a hand-edited report cannot mark a weak gate as
|
| 90 |
+
passed without satisfying its own configured test.
|
| 91 |
+
|
| 92 |
+
The SOTA validation configs also use `min_pair_win_rate` on improvement gates.
|
| 93 |
+
This is checked as wins divided by all paired samples, not only non-tied pairs.
|
| 94 |
+
It prevents an average-only improvement, or a single win with many ties, from
|
| 95 |
+
supporting a broad "better OCR/document engine" claim.
|
| 96 |
+
|
| 97 |
+
`required_validation_gates` is only valid on slots that also declare
|
| 98 |
+
`require_validation_status`. Each required gate baseline must also appear in
|
| 99 |
+
`require_baselines`, and each gate metric must appear in `required_metrics`.
|
| 100 |
+
|
| 101 |
+
For claim-grade slots, set `required_validation_config` to the checked-in
|
| 102 |
+
validation config path. `review-claim` hashes that file and requires
|
| 103 |
+
`experiment.json` to record the same `validation_config_sha256`. This prevents a
|
| 104 |
+
portfolio from passing with an easier local validation config that happens to
|
| 105 |
+
contain a subset of the expected gates.
|
| 106 |
+
|
| 107 |
+
For hybrid candidates, use `require_candidate_fallback_metrics`,
|
| 108 |
+
`max_candidate_fallback_trigger_rate`, and
|
| 109 |
+
`max_candidate_fallback_trigger_rate_by_slice`, and
|
| 110 |
+
`max_candidate_fallback_failed_rate`. These checks make fallback dependency
|
| 111 |
+
auditable. A candidate that wins mostly by routing pages to the fallback is a
|
| 112 |
+
different claim from a local-first engine with selective fallback, even if both
|
| 113 |
+
produce strong output metrics. The per-slice trigger limits matter because a
|
| 114 |
+
low aggregate fallback rate can still hide misuse on ordinary English or
|
| 115 |
+
Nepali text pages.
|
| 116 |
+
|
| 117 |
+
When an experiment records `environment.fallback_min_quality_score`,
|
| 118 |
+
`review-claim` also expects candidate rows to include
|
| 119 |
+
`fallback_primary_quality_score` and `fallback_min_quality_score` whenever
|
| 120 |
+
`require_candidate_fallback_metrics` is enabled. That keeps quality-gated
|
| 121 |
+
fallback policies auditable per sample instead of only recording the threshold
|
| 122 |
+
once in `experiment.json`.
|
| 123 |
+
|
| 124 |
+
Fallback model cards are audited separately from the primary candidate card.
|
| 125 |
+
When `experiment.json` records `fallback_model_config` and
|
| 126 |
+
`fallback_model_config_sha256`, `review-claim` verifies the hash and runs
|
| 127 |
+
`audit-model` on that fallback card. The SOTA profile also sets
|
| 128 |
+
`require_fallback_model_present: true`,
|
| 129 |
+
`require_candidate_fallback_provenance: true`,
|
| 130 |
+
`require_shared_fallback_model: true` and
|
| 131 |
+
`require_fallback_model_audit_passed: true`, so a portfolio cannot silently mix
|
| 132 |
+
unaudited fallback components across required experiment slots or omit fallback
|
| 133 |
+
provenance while still claiming the selective-fallback engine.
|
| 134 |
+
|
| 135 |
+
The candidate card must also declare the fallback component it was built to use:
|
| 136 |
+
|
| 137 |
+
```json
|
| 138 |
+
{
|
| 139 |
+
"provenance": {
|
| 140 |
+
"fallback_model_card": {
|
| 141 |
+
"path": "runs/fallback/model-card.json",
|
| 142 |
+
"sha256": "..."
|
| 143 |
+
}
|
| 144 |
+
}
|
| 145 |
+
}
|
| 146 |
+
```
|
| 147 |
+
|
| 148 |
+
`review-claim` hashes that model card, compares it with every reviewed
|
| 149 |
+
`fallback_model_config_sha256`, and audits the fallback card. This closes the
|
| 150 |
+
gap where an experiment could record a fallback at benchmark time while the
|
| 151 |
+
candidate card itself remained ambiguous about which selective-fallback engine
|
| 152 |
+
was being claimed.
|
| 153 |
+
|
| 154 |
+
That second guardrail matters for exactly the case we hit on `k2`: a rendered-document `script_select_composite` calibrated on `claimpack-rendered-v2` chose `primary_script_threshold=0.0`, meaning the route degenerated to "always pick Surya." Separation from the final holdout is not enough there; the candidate is still not a meaningful new routed model.
|
| 155 |
+
|
| 156 |
+
The holdout-equivalence guardrail covers the next failure mode after calibration separation: even when calibration was done on a separate split, a routed candidate can still reduce to one component baseline on the reviewed holdout. If the paired benchmark rows show `0` wins, `0` losses, and all ties on CER against a component baseline, `review-claim` now fails `candidate-distinctness`.
|
| 157 |
+
|
| 158 |
+
For the SOTA track, set `require_trained_recognizer: true`. A direct
|
| 159 |
+
`paddleocr` or `hf_vision_encoder_decoder` candidate passes this check only when
|
| 160 |
+
the model card has recorded artifacts plus `provenance.train_manifest`,
|
| 161 |
+
`provenance.eval_manifest`, and `provenance.training_summary` with SHA-256
|
| 162 |
+
hashes. Review resolves those paths from the model-card location and verifies
|
| 163 |
+
the files still match the recorded hashes. Composite candidates must point to
|
| 164 |
+
the trained component with:
|
| 165 |
+
|
| 166 |
+
```json
|
| 167 |
+
{
|
| 168 |
+
"provenance": {
|
| 169 |
+
"trained_recognizer_model_card": {
|
| 170 |
+
"path": "runs/recognizer/package/model-card.json",
|
| 171 |
+
"sha256": "..."
|
| 172 |
+
}
|
| 173 |
+
}
|
| 174 |
+
}
|
| 175 |
+
```
|
| 176 |
+
|
| 177 |
+
This prevents a wrapper around Tesseract, Surya, or a VLM from satisfying the
|
| 178 |
+
paper objective by benchmark routing alone.
|
| 179 |
+
|
| 180 |
+
For recognizer branch admission, set `require_model_admission: true` and record
|
| 181 |
+
the admission validation report in the model card:
|
| 182 |
+
|
| 183 |
+
```json
|
| 184 |
+
{
|
| 185 |
+
"metrics": {
|
| 186 |
+
"admission_validation": {
|
| 187 |
+
"path": "outputs/validation-next-recognizer-hard50/validation.json",
|
| 188 |
+
"sha256": "...",
|
| 189 |
+
"required_status": "validated"
|
| 190 |
+
}
|
| 191 |
+
}
|
| 192 |
+
}
|
| 193 |
+
```
|
| 194 |
+
|
| 195 |
+
This catches the failure mode we hit with the PaddleOCR branches: the model can
|
| 196 |
+
have artifacts and train/eval provenance, but still be empirically rejected by
|
| 197 |
+
the hard recognizer gate. Rejected admission reports are summarized with their
|
| 198 |
+
failed requirements and gate deltas, so review output can show why a recognizer
|
| 199 |
+
was rejected without manually opening the raw validation JSON. Admission reports
|
| 200 |
+
are also bound back to `provenance.candidate_model_config`; review rejects a
|
| 201 |
+
passing admission artifact if it points at a different model id, backend, base
|
| 202 |
+
model, or artifact set.
|
| 203 |
+
|
| 204 |
+
For document-engine SOTA claims, metric presence is not enough. The
|
| 205 |
+
`configs/claim-review-sota-v1.json` portfolio requires passing validation gates
|
| 206 |
+
for the named claim dimensions:
|
| 207 |
+
|
| 208 |
+
- OCR: CER and WER gates on Nepali/English slices where applicable
|
| 209 |
+
- tables: `table_cell_f1`
|
| 210 |
+
- reading order: `reading_order_pair_accuracy`
|
| 211 |
+
- figures: `figure_detection_f1` and `figure_caption_cer`
|
| 212 |
+
- reconstruction: `document_reconstruction_similarity`
|
| 213 |
+
|
| 214 |
+
The corresponding validation configs must contain matching gates. The
|
| 215 |
+
`tests/test_claim_review_sota_config.py` checks pin this relationship so a
|
| 216 |
+
future config edit cannot silently downgrade figure/caption or WER from a
|
| 217 |
+
passing gate to a reported-only metric.
|
| 218 |
+
|
| 219 |
+
The SOTA profile also requires exported structured-output artifacts from the
|
| 220 |
+
candidate benchmark rows:
|
| 221 |
+
|
| 222 |
+
- every required slot: `document.json`, `document.md`, and `quality.json`
|
| 223 |
+
- table slices: table CSV and HTML exports under `tables/`
|
| 224 |
+
- figure slices: `figures/metadata.json` and exported figure image files
|
| 225 |
+
|
| 226 |
+
For current benchmark reports, `review-claim` uses the candidate row's
|
| 227 |
+
`output_artifacts` records and verifies each recorded path, SHA-256 hash, and
|
| 228 |
+
byte size before inspecting content. Older reports without `output_artifacts`
|
| 229 |
+
fall back to path inference from the benchmark report directory and candidate
|
| 230 |
+
row input stem. JSON artifacts must parse with the expected top-level shape,
|
| 231 |
+
Markdown/table/image files must be nonempty, and figure metadata must contain
|
| 232 |
+
entries. This means a claim cannot be supported only by scalar metrics, stale
|
| 233 |
+
files, or swapped artifacts if the human-readable Markdown, machine-readable
|
| 234 |
+
JSON, tables, or figures were not actually produced as usable artifacts by the
|
| 235 |
+
reviewed benchmark row.
|
| 236 |
+
|
| 237 |
+
## Example Config
|
| 238 |
+
|
| 239 |
+
```json
|
| 240 |
+
{
|
| 241 |
+
"require_trained_recognizer": true,
|
| 242 |
+
"require_model_admission": true,
|
| 243 |
+
"require_fallback_model_present": true,
|
| 244 |
+
"require_candidate_fallback_provenance": true,
|
| 245 |
+
"require_shared_fallback_model": true,
|
| 246 |
+
"require_fallback_model_audit_passed": true,
|
| 247 |
+
"required_experiments": [
|
| 248 |
+
{
|
| 249 |
+
"name": "internal-nepali-pack",
|
| 250 |
+
"datasets": ["ocrtech-eval-pack"],
|
| 251 |
+
"require_validation_status": "validated",
|
| 252 |
+
"require_baselines": ["candidate", "tesseract", "stock-paddle", "glm-ocr", "paddleocr-vl"],
|
| 253 |
+
"required_metrics": ["cer", "wer", "document_reconstruction_similarity", "table_cell_f1", "reading_order_pair_accuracy", "figure_detection_f1", "figure_caption_cer"],
|
| 254 |
+
"required_metric_slices": {
|
| 255 |
+
"cer": ["nepali", "english"],
|
| 256 |
+
"wer": ["nepali", "english"],
|
| 257 |
+
"document_reconstruction_similarity": ["table", "reading_order", "figure"],
|
| 258 |
+
"table_cell_f1": ["table"],
|
| 259 |
+
"reading_order_pair_accuracy": ["reading_order"],
|
| 260 |
+
"figure_detection_f1": ["figure"],
|
| 261 |
+
"figure_caption_cer": ["figure"]
|
| 262 |
+
},
|
| 263 |
+
"required_validation_gates": [
|
| 264 |
+
{"baseline": "tesseract", "metric": "cer", "slices": ["nepali"]},
|
| 265 |
+
{"baseline": "stock-paddle", "metric": "cer", "slices": ["nepali"]},
|
| 266 |
+
{"baseline": "glm-ocr", "metric": "cer", "slices": ["nepali"]},
|
| 267 |
+
{"baseline": "surya", "metric": "table_cell_f1", "slices": ["table"]},
|
| 268 |
+
{"baseline": "paddleocr-vl", "metric": "reading_order_pair_accuracy", "slices": ["reading_order"]}
|
| 269 |
+
],
|
| 270 |
+
"require_candidate_fallback_metrics": true,
|
| 271 |
+
"max_candidate_fallback_trigger_rate": 0.35,
|
| 272 |
+
"max_candidate_fallback_failed_rate": 0.0,
|
| 273 |
+
"required_slices": ["nepali", "english", "table", "reading_order", "figure", "scan"]
|
| 274 |
+
},
|
| 275 |
+
{
|
| 276 |
+
"name": "ocrbench-v2",
|
| 277 |
+
"datasets": ["ocrbench-v2"],
|
| 278 |
+
"require_validation_status": "validated",
|
| 279 |
+
"required_metrics": ["cer", "wer"],
|
| 280 |
+
"require_baselines": ["candidate", "glm-ocr", "paddleocr-vl"]
|
| 281 |
+
},
|
| 282 |
+
{
|
| 283 |
+
"name": "omnidocbench",
|
| 284 |
+
"datasets": ["omnidocbench"],
|
| 285 |
+
"require_validation_status": "validated",
|
| 286 |
+
"required_metrics": ["cer", "document_reconstruction_similarity", "table_cell_f1", "reading_order_pair_accuracy"],
|
| 287 |
+
"require_baselines": ["candidate", "paddleocr-vl"]
|
| 288 |
+
}
|
| 289 |
+
]
|
| 290 |
+
}
|
| 291 |
+
```
|
| 292 |
+
|
| 293 |
+
The config is intentionally explicit. If two experiments match one slot, the review fails. Tighten the config until every required slot maps to one concrete artifact.
|
| 294 |
+
`eval_manifest_patterns` use shell-style wildcards, so a slot such as
|
| 295 |
+
`"*Real5-OmniDocBench*"` matches a manifest path containing that benchmark name.
|
| 296 |
+
|
| 297 |
+
The checked-in SOTA profile is paired with slot-specific validation configs:
|
| 298 |
+
|
| 299 |
+
- `configs/validation-internal-sota-v1.json`
|
| 300 |
+
- `configs/validation-public-omnidocbench-sota-v1.json`
|
| 301 |
+
- `configs/validation-public-real5-sota-v1.json`
|
| 302 |
+
|
| 303 |
+
Those configs are expected to produce the passed gates referenced by
|
| 304 |
+
`required_validation_gates`.
|
| 305 |
+
|
| 306 |
+
## Run
|
| 307 |
+
|
| 308 |
+
```bash
|
| 309 |
+
ocrtech review-claim \
|
| 310 |
+
--experiment outputs/internal-claim/experiment.json \
|
| 311 |
+
--experiment outputs/ocrbench-v2/experiment.json \
|
| 312 |
+
--experiment outputs/omnidocbench/experiment.json \
|
| 313 |
+
--config claim-review.json \
|
| 314 |
+
--model-config runs/recognizer/package/model-card.json \
|
| 315 |
+
--out outputs/claim-review
|
| 316 |
+
```
|
| 317 |
+
|
| 318 |
+
## Outputs
|
| 319 |
+
|
| 320 |
+
```text
|
| 321 |
+
outputs/claim-review/
|
| 322 |
+
claim-review.json
|
| 323 |
+
claim-review.md
|
| 324 |
+
```
|
| 325 |
+
|
| 326 |
+
If this report fails, the claim is not review-ready.
|
| 327 |
+
|
| 328 |
+
Do not hand-edit copied `experiment.json` files to point at convenient local
|
| 329 |
+
paths. If an experiment was run remotely, sync the benchmark report, manifest,
|
| 330 |
+
references, model card, and validation report together, or regenerate the
|
| 331 |
+
experiment report from the synced paths. `review-claim` intentionally fails on
|
| 332 |
+
stale paths instead of guessing because path rewrites without hashes destroy
|
| 333 |
+
claim provenance.
|
ocr-tech-mps-integration/docs/CLAUDE_HANDOFF.md
ADDED
|
@@ -0,0 +1,613 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Claude Handoff: Nepal Multilingual OCR
|
| 2 |
+
|
| 3 |
+
Goal: build and publish a research-grade PaddleOCR/PP-Structure OCR engine for
|
| 4 |
+
at least ten languages spoken in Nepal, using Gorkhapatra/Naya Nepal pages as
|
| 5 |
+
real validation evidence, PDF-native and legacy-font extraction for weak labels,
|
| 6 |
+
and script-aware synthetic data for training and ablation.
|
| 7 |
+
|
| 8 |
+
Updated: 2026-06-16
|
| 9 |
+
Repo: `Ampixa/ocr-tech`
|
| 10 |
+
Visibility: private at handoff time
|
| 11 |
+
Branch: `main`
|
| 12 |
+
|
| 13 |
+
This handoff is for a follow-on Claude/Codex agent. The user is steering a
|
| 14 |
+
research-grade OCR project, not a demo. The target is a PaddleOCR-centered
|
| 15 |
+
document OCR engine whose claims are tied to real validation and reproducible
|
| 16 |
+
training data, not synthetic-only benchmarks.
|
| 17 |
+
|
| 18 |
+
## Operating Rules
|
| 19 |
+
|
| 20 |
+
- Use `uv run ...` for project commands.
|
| 21 |
+
- Use `apply_patch` for manual edits.
|
| 22 |
+
- Do not commit generated corpora, rendered images, model runs, or outputs
|
| 23 |
+
unless explicitly requested; `data/`, `outputs/`, and `runs/` are ignored
|
| 24 |
+
intentionally.
|
| 25 |
+
- Do commit reproducibility configs, docs, tests, and source code changes.
|
| 26 |
+
- GPG signing fails on this machine; use:
|
| 27 |
+
|
| 28 |
+
```bash
|
| 29 |
+
git -c commit.gpgsign=false commit -m "<message>"
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
- Do not commit remote passwords or credentials. Hosts are documented below,
|
| 33 |
+
but credentials must stay in the conversation/session, not tracked files.
|
| 34 |
+
- Do not claim SOTA or model quality unless backed by a passed real validation
|
| 35 |
+
audit and benchmark.
|
| 36 |
+
- Synthetic OCR data is never claim evidence. It is training/ablation material.
|
| 37 |
+
- Validate schemas before writing converters. Do not infer fields from file
|
| 38 |
+
names.
|
| 39 |
+
|
| 40 |
+
## Current Research Goal
|
| 41 |
+
|
| 42 |
+
Build a PaddleOCR/PP-OCR + PP-Structure-style document engine for at least ten
|
| 43 |
+
languages spoken in Nepal.
|
| 44 |
+
|
| 45 |
+
Core scope:
|
| 46 |
+
|
| 47 |
+
- OCR recognition
|
| 48 |
+
- reading order
|
| 49 |
+
- tables
|
| 50 |
+
- figures/captions
|
| 51 |
+
- Markdown output
|
| 52 |
+
- JSON output
|
| 53 |
+
- optional VLM fallback for hard figures/pages
|
| 54 |
+
|
| 55 |
+
Current research angle:
|
| 56 |
+
|
| 57 |
+
> Low-resource Nepal language OCR adaptation using PDF-native weak labels,
|
| 58 |
+
> legacy font conversion, real Gorkhapatra validation, and script-aware
|
| 59 |
+
> synthetic robustness.
|
| 60 |
+
|
| 61 |
+
The real claim should be about measurable OCR/structure improvement on held-out
|
| 62 |
+
Gorkhapatra/Naya Nepal pages, not about "understanding everything."
|
| 63 |
+
|
| 64 |
+
## Repo State
|
| 65 |
+
|
| 66 |
+
Important recent commits:
|
| 67 |
+
|
| 68 |
+
```text
|
| 69 |
+
22454e2 Document cdjk language corpus migration
|
| 70 |
+
1f98430 Record migrated synthesis text sweep
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
The repo was checked with:
|
| 74 |
+
|
| 75 |
+
```bash
|
| 76 |
+
jq . configs/cdjk-language-corpus-migration-v1.json >/dev/null
|
| 77 |
+
git diff --check
|
| 78 |
+
uv run pytest tests/test_synthesis_resources.py -q
|
| 79 |
+
```
|
| 80 |
+
|
| 81 |
+
All passed before this handoff was written.
|
| 82 |
+
|
| 83 |
+
At handoff time, the only unrelated visible untracked path was:
|
| 84 |
+
|
| 85 |
+
```text
|
| 86 |
+
.playwright-mcp/
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
That is browser scratch output and should not be committed.
|
| 90 |
+
|
| 91 |
+
## Most Important Existing Docs
|
| 92 |
+
|
| 93 |
+
Read these before acting:
|
| 94 |
+
|
| 95 |
+
- `docs/ACTIVE_RESEARCH_CHECKLIST.md`
|
| 96 |
+
- `docs/GORKHAPATRA_LANGUAGE_PAGE_ALIGNMENT.md`
|
| 97 |
+
- `docs/CDJK_LANGUAGE_CORPUS_MIGRATION.md`
|
| 98 |
+
- `docs/SYNTHESIS_RESOURCE_AUDIT.md`
|
| 99 |
+
- `docs/FONT_ASSETS.md`
|
| 100 |
+
- `docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md`
|
| 101 |
+
- `docs/NEPAL_MULTILINGUAL_OCR_GOAL.md`
|
| 102 |
+
|
| 103 |
+
## What Has Been Built
|
| 104 |
+
|
| 105 |
+
### CLI Surface
|
| 106 |
+
|
| 107 |
+
The repo already has many `ocrtech` commands. Most relevant now:
|
| 108 |
+
|
| 109 |
+
```text
|
| 110 |
+
ocrtech audit-gorkhapatra-language-pages
|
| 111 |
+
ocrtech prepare-gorkhapatra-language-page-pack
|
| 112 |
+
ocrtech prepare-gorkhapatra-language-page-review
|
| 113 |
+
ocrtech audit-gorkhapatra-language-page-review
|
| 114 |
+
ocrtech extract-gorkhapatra-language-page-pdf-text
|
| 115 |
+
ocrtech convert-limbu-legacy
|
| 116 |
+
ocrtech audit-synthesis-resources
|
| 117 |
+
ocrtech prepare-limbu-limdic-text
|
| 118 |
+
ocrtech prepare-limbu-unicode-text
|
| 119 |
+
ocrtech prepare-tamang-text
|
| 120 |
+
ocrtech prepare-magar-text
|
| 121 |
+
ocrtech prepare-toolkit-parallel-text
|
| 122 |
+
ocrtech audit-synthesis-text-manifest
|
| 123 |
+
ocrtech split-synthesis-text
|
| 124 |
+
ocrtech render-synthesis-text-lines
|
| 125 |
+
ocrtech render-synthesis-text-split
|
| 126 |
+
ocrtech audit-rendered-synthesis-lines
|
| 127 |
+
ocrtech audit-rendered-degradation-split
|
| 128 |
+
ocrtech train-recognizer
|
| 129 |
+
```
|
| 130 |
+
|
| 131 |
+
### Limbu PDF-Native Extraction
|
| 132 |
+
|
| 133 |
+
Source rule for every Gorkhapatra/Naya Nepal language:
|
| 134 |
+
`https://gorkhapatraonline.com/publications/naya-nepal` is only the discovery
|
| 135 |
+
index for identifying which language/date was posted. Do not use the Naya Nepal
|
| 136 |
+
blog/card image as OCR, eval, or training evidence. Fetch the full paper from
|
| 137 |
+
the Gorkhapatra epaper date-range endpoint, for example
|
| 138 |
+
`https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-16&pdfdaterange2=2026-06-23&slug=gorkhapatra`,
|
| 139 |
+
then render the matching PDF page and extract/convert the text layer from that
|
| 140 |
+
same PDF page. Record PDF URL/hash/page number, rendered image hash, text
|
| 141 |
+
extraction provenance, and conversion map hash. Classify each language page as
|
| 142 |
+
`unicode_native`, `legacy_font`, `devanagari_transcription`, `latin_notation`, or
|
| 143 |
+
`no_text_layer`; manual transcription/native-reader work is only the fallback
|
| 144 |
+
when the PDF has no usable text layer or the font cannot yet be decoded.
|
| 145 |
+
|
| 146 |
+
The current successful Gorkhapatra Limbu path:
|
| 147 |
+
|
| 148 |
+
```bash
|
| 149 |
+
uv run ocrtech extract-gorkhapatra-language-page-pdf-text \
|
| 150 |
+
data/limbu-gorkhapatra-language-page-pack-2026-06-16/manifests/gorkhapatra-language-page-pack.jsonl \
|
| 151 |
+
--candidate-language लिम्बू \
|
| 152 |
+
--out data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16
|
| 153 |
+
```
|
| 154 |
+
|
| 155 |
+
Known result:
|
| 156 |
+
|
| 157 |
+
```text
|
| 158 |
+
passed=True samples=2/2 spans=776 target_spans=304 fonts=10 warnings=0
|
| 159 |
+
sample_id=gorkhapatra-language-page-0001-01
|
| 160 |
+
pdf_page_number=6
|
| 161 |
+
target_font=NamdhinggoSILL
|
| 162 |
+
target_spans=304
|
| 163 |
+
embedded_font=UFNWLJ+NamdhinggoSILL
|
| 164 |
+
```
|
| 165 |
+
|
| 166 |
+
Page conclusion:
|
| 167 |
+
|
| 168 |
+
- page 6 is the Limbu/Sirijonga page
|
| 169 |
+
- page 7 is the same-date non-Limbu page
|
| 170 |
+
|
| 171 |
+
The PDF text layer exposes legacy ASCII-like Namdhinggo text, not Unicode
|
| 172 |
+
Limbu. Example:
|
| 173 |
+
|
| 174 |
+
```text
|
| 175 |
+
legacy: kfMG g' ;fK;SF[ yf]af]cf
|
| 176 |
+
unicode: ᤐᤠ᤺ᤴ ᤏᤢ ᤛᤠᤵᤛᤰᤖᤧ ᤌᤥᤒᤥᤀᤠ
|
| 177 |
+
```
|
| 178 |
+
|
| 179 |
+
The converter is implemented in `src/ocrtech/limbu_legacy.py` using the vendored
|
| 180 |
+
SIL map:
|
| 181 |
+
|
| 182 |
+
```text
|
| 183 |
+
data/mappings/limbu-legacy/Limbu.map
|
| 184 |
+
```
|
| 185 |
+
|
| 186 |
+
### Limbu Review Dashboard
|
| 187 |
+
|
| 188 |
+
Generated dashboard path:
|
| 189 |
+
|
| 190 |
+
```text
|
| 191 |
+
data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html
|
| 192 |
+
```
|
| 193 |
+
|
| 194 |
+
Correct local file URL:
|
| 195 |
+
|
| 196 |
+
```text
|
| 197 |
+
file:///Users/cdjk/github/llm/ocr-tech/data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html
|
| 198 |
+
```
|
| 199 |
+
|
| 200 |
+
This is the correct browser UI for the Gorkhapatra PDF-native Limbu page. It
|
| 201 |
+
shows the full rendered PDF image with bounded boxes; clicking a box selects the
|
| 202 |
+
matching right-side row with converted text and the corresponding rendered span
|
| 203 |
+
crop. Do not substitute the PDF-native verification-bundle HTML for this task;
|
| 204 |
+
that was the wrong/broken UI.
|
| 205 |
+
|
| 206 |
+
Local server used:
|
| 207 |
+
|
| 208 |
+
```bash
|
| 209 |
+
uv run python -m http.server 8765 --bind 127.0.0.1
|
| 210 |
+
```
|
| 211 |
+
|
| 212 |
+
Dashboard URL:
|
| 213 |
+
|
| 214 |
+
```text
|
| 215 |
+
http://127.0.0.1:8765/data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html
|
| 216 |
+
```
|
| 217 |
+
|
| 218 |
+
Important bug already fixed in the generated dashboard: clicking a bounding box
|
| 219 |
+
must not re-render or scroll the image/page back to the top. The fix removed
|
| 220 |
+
automatic `scrollIntoView` from bounding-box selection.
|
| 221 |
+
|
| 222 |
+
If this dashboard is regenerated by code later, preserve this behavior:
|
| 223 |
+
|
| 224 |
+
```js
|
| 225 |
+
function selectSpan(id){
|
| 226 |
+
selected=id;
|
| 227 |
+
document.querySelectorAll('.box.selected,.span.selected').forEach(el => el.classList.remove('selected'));
|
| 228 |
+
document.querySelectorAll(`[data-span="${id}"],[data-row="${id}"]`).forEach(el => el.classList.add('selected'));
|
| 229 |
+
}
|
| 230 |
+
```
|
| 231 |
+
|
| 232 |
+
### Migrated Corpus
|
| 233 |
+
|
| 234 |
+
Source host:
|
| 235 |
+
|
| 236 |
+
```text
|
| 237 |
+
cdjk@100.110.148.43
|
| 238 |
+
machine: cdjks-MacBook-Pro.local
|
| 239 |
+
external volume: /Volumes/TRANSCEND
|
| 240 |
+
```
|
| 241 |
+
|
| 242 |
+
Do not write generated data back to that host. Treat it as a source for corpus
|
| 243 |
+
and dictionaries unless the user explicitly asks otherwise.
|
| 244 |
+
|
| 245 |
+
Tracked migration policy:
|
| 246 |
+
|
| 247 |
+
```text
|
| 248 |
+
configs/cdjk-language-corpus-migration-v1.json
|
| 249 |
+
```
|
| 250 |
+
|
| 251 |
+
Ignored local copy:
|
| 252 |
+
|
| 253 |
+
```text
|
| 254 |
+
data/external-language-resources/cdjk-100.110.148.43/
|
| 255 |
+
```
|
| 256 |
+
|
| 257 |
+
Copied roots:
|
| 258 |
+
|
| 259 |
+
```text
|
| 260 |
+
dhimal-toolkit
|
| 261 |
+
gurung-language-toolkit
|
| 262 |
+
limbu-speech-toolkit
|
| 263 |
+
ltk
|
| 264 |
+
magar-toolkit
|
| 265 |
+
rai-speech-toolkit
|
| 266 |
+
```
|
| 267 |
+
|
| 268 |
+
Migration result:
|
| 269 |
+
|
| 270 |
+
```text
|
| 271 |
+
2307 text/resource files transferred
|
| 272 |
+
1.6G local size
|
| 273 |
+
resource audit: passed=True roots=6/6 files=2308 warnings=0
|
| 274 |
+
```
|
| 275 |
+
|
| 276 |
+
### Prepared Synthesis Text
|
| 277 |
+
|
| 278 |
+
Generated synthesis text is ignored under `data/synthesis-text/`, but the
|
| 279 |
+
documented state is:
|
| 280 |
+
|
| 281 |
+
```text
|
| 282 |
+
20 manifests
|
| 283 |
+
228212 total samples
|
| 284 |
+
45258 duplicate rows removed/reported by converters
|
| 285 |
+
202560 Devanagari rows
|
| 286 |
+
10337 Mixed_Devanagari_Latin rows
|
| 287 |
+
6521 Latin rows
|
| 288 |
+
8772 true Unicode Limbu rows
|
| 289 |
+
14 Mixed_Limbu_Devanagari rows
|
| 290 |
+
3 Mixed_Limbu_Latin rows
|
| 291 |
+
5 Other rows
|
| 292 |
+
```
|
| 293 |
+
|
| 294 |
+
Important generated manifests:
|
| 295 |
+
|
| 296 |
+
```text
|
| 297 |
+
data/synthesis-text/cdjk-limbu-limdic-2026-06-16/limbu-limdic-text.jsonl
|
| 298 |
+
data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered/limbu-only.jsonl
|
| 299 |
+
data/synthesis-text/cdjk-tamang-2026-06-16/tamang-text.jsonl
|
| 300 |
+
data/synthesis-text/cdjk-magar-2026-06-16/magar-text.jsonl
|
| 301 |
+
data/synthesis-text/cdjk-gurung-2026-06-16/toolkit-parallel-text.jsonl
|
| 302 |
+
data/synthesis-text/cdjk-dhimal-parallel-2026-06-16/toolkit-parallel-text.jsonl
|
| 303 |
+
data/synthesis-text/cdjk-dhimal-mono-2026-06-16/toolkit-parallel-text.jsonl
|
| 304 |
+
data/synthesis-text/cdjk-magar-parallel-extra-2026-06-16/toolkit-parallel-text.jsonl
|
| 305 |
+
data/synthesis-text/cdjk-magar-mono-extra-2026-06-16/toolkit-parallel-text.jsonl
|
| 306 |
+
data/synthesis-text/cdjk-rai-*-2026-06-16/toolkit-parallel-text.jsonl
|
| 307 |
+
```
|
| 308 |
+
|
| 309 |
+
All synthesis text audits passed with `claim_eligible=0`.
|
| 310 |
+
|
| 311 |
+
Important interpretation:
|
| 312 |
+
|
| 313 |
+
- True Unicode Limbu exists from Limdic and should render with Namdhinggo after
|
| 314 |
+
filtering `script == Limbu`.
|
| 315 |
+
- Most Gurung, Dhimal, Magar, Tamang, and Rai rows are Devanagari-transcribed
|
| 316 |
+
language text, not native-script OCR.
|
| 317 |
+
- Bantawa, Khaling, Koyu, and Thulung include substantial Latin phonetic data.
|
| 318 |
+
These are useful for language resources but should not be called native-script
|
| 319 |
+
OCR data.
|
| 320 |
+
|
| 321 |
+
### Synthetic Rendering
|
| 322 |
+
|
| 323 |
+
Corrected Limbu-only smoke render:
|
| 324 |
+
|
| 325 |
+
```bash
|
| 326 |
+
uv run ocrtech render-synthesis-text-lines \
|
| 327 |
+
data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered/limbu-only.jsonl \
|
| 328 |
+
--out data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only \
|
| 329 |
+
--font-path data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-Regular.ttf \
|
| 330 |
+
--limit 120 \
|
| 331 |
+
--degradation-profile clean \
|
| 332 |
+
--degradation-profile scan \
|
| 333 |
+
--degradation-profile low_light \
|
| 334 |
+
--degradation-profile uneven_light \
|
| 335 |
+
--degradation-profile camera_left \
|
| 336 |
+
--degradation-profile phone_photo
|
| 337 |
+
```
|
| 338 |
+
|
| 339 |
+
Audit:
|
| 340 |
+
|
| 341 |
+
```bash
|
| 342 |
+
uv run ocrtech audit-rendered-synthesis-lines \
|
| 343 |
+
data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only/manifests/rendered-line-manifest.jsonl \
|
| 344 |
+
--label-file data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only/labels/train.txt \
|
| 345 |
+
--out outputs/cdjk-limbu-rendered-smoke-audit-2026-06-16-limbu-only
|
| 346 |
+
```
|
| 347 |
+
|
| 348 |
+
Result:
|
| 349 |
+
|
| 350 |
+
```text
|
| 351 |
+
passed=True samples=120 claim_eligible=0 issues=0 warnings=0
|
| 352 |
+
degradation_counts:
|
| 353 |
+
camera_left: 20
|
| 354 |
+
clean: 20
|
| 355 |
+
low_light: 20
|
| 356 |
+
phone_photo: 20
|
| 357 |
+
scan: 20
|
| 358 |
+
uneven_light: 20
|
| 359 |
+
```
|
| 360 |
+
|
| 361 |
+
Important lesson: an earlier mixed-script render produced tofu boxes because
|
| 362 |
+
Devanagari/Latin rows were rendered with a Limbu font. Always filter by script
|
| 363 |
+
before choosing a font.
|
| 364 |
+
|
| 365 |
+
## Current Strategic Decision
|
| 366 |
+
|
| 367 |
+
The user asked what next. The right order is:
|
| 368 |
+
|
| 369 |
+
1. Build Gorkhapatra validation for all target languages.
|
| 370 |
+
2. Create language/script/font matrix from actual Gorkhapatra PDF evidence.
|
| 371 |
+
3. Generate split-safe synthetic data per script/font bucket.
|
| 372 |
+
4. Research and execute PaddleOCR recognizer fine-tuning.
|
| 373 |
+
5. Keep PP-Structure/layout mostly stock until recognizer evidence says
|
| 374 |
+
detection/layout fine-tuning is required.
|
| 375 |
+
|
| 376 |
+
Do not scale synthetic data first without real validation. It will create a
|
| 377 |
+
large training set but no credible paper claim.
|
| 378 |
+
|
| 379 |
+
## Immediate Next Task
|
| 380 |
+
|
| 381 |
+
Build a reusable all-language Gorkhapatra validation generator.
|
| 382 |
+
|
| 383 |
+
It should turn this manual Limbu success into a repeatable command/workflow:
|
| 384 |
+
|
| 385 |
+
```text
|
| 386 |
+
Naya Nepal publication card
|
| 387 |
+
-> date/language/article metadata
|
| 388 |
+
-> matching epaper PDF
|
| 389 |
+
-> candidate language-page pages
|
| 390 |
+
-> rendered page image
|
| 391 |
+
-> PDF-native spans/fonts/bboxes/codepoints
|
| 392 |
+
-> optional legacy conversion by script/font
|
| 393 |
+
-> review dashboard
|
| 394 |
+
-> reviewed JSON/CSV annotations
|
| 395 |
+
```
|
| 396 |
+
|
| 397 |
+
Suggested CLI shape:
|
| 398 |
+
|
| 399 |
+
```text
|
| 400 |
+
ocrtech prepare-gorkhapatra-language-validation
|
| 401 |
+
```
|
| 402 |
+
|
| 403 |
+
Likely arguments:
|
| 404 |
+
|
| 405 |
+
```text
|
| 406 |
+
--language-label <Nepali label>
|
| 407 |
+
--publication-pages <N>
|
| 408 |
+
--epaper-url <date-range endpoint>
|
| 409 |
+
--out <data/gorkhapatra-validation-...>
|
| 410 |
+
--target-font <optional font name>
|
| 411 |
+
--legacy-converter <optional converter id, e.g. limbu-namdhinggo>
|
| 412 |
+
--render-pages
|
| 413 |
+
--download-pdfs
|
| 414 |
+
```
|
| 415 |
+
|
| 416 |
+
Output should include:
|
| 417 |
+
|
| 418 |
+
```text
|
| 419 |
+
validation.json
|
| 420 |
+
validation.md
|
| 421 |
+
samples/*.pdf-text.json
|
| 422 |
+
samples/*.pdf-text.txt
|
| 423 |
+
spans.csv
|
| 424 |
+
fonts/
|
| 425 |
+
page-images/
|
| 426 |
+
span-renders/
|
| 427 |
+
dashboard.html
|
| 428 |
+
review.csv
|
| 429 |
+
review.jsonl
|
| 430 |
+
```
|
| 431 |
+
|
| 432 |
+
Dashboard requirements:
|
| 433 |
+
|
| 434 |
+
- Full page image with clickable bbox overlay.
|
| 435 |
+
- Right panel with selected span, PDF-rendered crop, raw extracted text,
|
| 436 |
+
converted Unicode text when available, font, bbox, codepoints.
|
| 437 |
+
- Review status controls:
|
| 438 |
+
- `accept`
|
| 439 |
+
- `needs_fix`
|
| 440 |
+
- `bad_segment`
|
| 441 |
+
- `drop`
|
| 442 |
+
- `wrong_page`
|
| 443 |
+
- `wrong_language`
|
| 444 |
+
- Must preserve image/page scroll when clicking bbox.
|
| 445 |
+
- Must not ask the user to transcribe text if PDF-native extraction/conversion
|
| 446 |
+
already produces usable Unicode text.
|
| 447 |
+
- Must save/export annotations as JSONL/CSV.
|
| 448 |
+
|
| 449 |
+
## Language Validation Targets
|
| 450 |
+
|
| 451 |
+
At least ten target languages are desired. Current likely sources:
|
| 452 |
+
|
| 453 |
+
- Limbu/Sirijonga
|
| 454 |
+
- Tamang
|
| 455 |
+
- Magar
|
| 456 |
+
- Gurung
|
| 457 |
+
- Dhimal
|
| 458 |
+
- Newar/Newa
|
| 459 |
+
- Bantawa/Rai
|
| 460 |
+
- Chamling/Rai
|
| 461 |
+
- Kulung/Rai
|
| 462 |
+
- Athpariya/Rai
|
| 463 |
+
- Khaling/Rai
|
| 464 |
+
- Yakkha/Rai
|
| 465 |
+
- Sunwar/Rai
|
| 466 |
+
- Lohorung/Rai
|
| 467 |
+
- Dumi/Rai
|
| 468 |
+
- Thulung/Rai
|
| 469 |
+
- Koyu/Rai
|
| 470 |
+
|
| 471 |
+
But Gorkhapatra validation must decide actual script form:
|
| 472 |
+
|
| 473 |
+
- Unicode native script
|
| 474 |
+
- legacy font encoding
|
| 475 |
+
- Devanagari transcription
|
| 476 |
+
- Latin notation
|
| 477 |
+
- unsupported/no text layer
|
| 478 |
+
|
| 479 |
+
Do not assume native Unicode just because the language has a script.
|
| 480 |
+
|
| 481 |
+
## Training Research Direction
|
| 482 |
+
|
| 483 |
+
Use primary sources before changing training code. Current understanding:
|
| 484 |
+
|
| 485 |
+
- PaddleOCR/PP-OCR remains the primary engine family.
|
| 486 |
+
- Start by fine-tuning recognizer, not layout/detection.
|
| 487 |
+
- Use PP-Structure/PP-StructureV3-style parsing for document layout and
|
| 488 |
+
structured output.
|
| 489 |
+
- Consider PaddleOCR-VL only as fallback/teacher/comparator, not the default
|
| 490 |
+
small local engine.
|
| 491 |
+
|
| 492 |
+
Recognizer-first rationale:
|
| 493 |
+
|
| 494 |
+
- The repo already has line/synthesis text and rendered line labels.
|
| 495 |
+
- Recognition turns cropped text images into text.
|
| 496 |
+
- Detection/layout claims require bbox/layout ground truth, which is not yet
|
| 497 |
+
mature across languages.
|
| 498 |
+
- Fine-tuning detection before measuring detection failures is premature.
|
| 499 |
+
|
| 500 |
+
Training prerequisites before a serious run:
|
| 501 |
+
|
| 502 |
+
- Split synthesis text with `split-synthesis-text`.
|
| 503 |
+
- Exclude real validation/reference text hashes from synthetic training.
|
| 504 |
+
- Render train/eval with font readiness and degradation audits.
|
| 505 |
+
- Keep English/Nepali non-regression sets.
|
| 506 |
+
- Package PaddleOCR recognizer training recipe.
|
| 507 |
+
- Evaluate on Gorkhapatra real pages and held-out synthetic rows.
|
| 508 |
+
|
| 509 |
+
## Remote Machines
|
| 510 |
+
|
| 511 |
+
Known hosts from the session:
|
| 512 |
+
|
| 513 |
+
```text
|
| 514 |
+
k2@100.97.238.103
|
| 515 |
+
cdjk@100.110.148.43
|
| 516 |
+
cdjk@100.117.21.47
|
| 517 |
+
```
|
| 518 |
+
|
| 519 |
+
Do not commit passwords. The user provided credentials in chat previously, but
|
| 520 |
+
tracked docs must not contain them.
|
| 521 |
+
|
| 522 |
+
Important storage findings:
|
| 523 |
+
|
| 524 |
+
- `cdjk@100.110.148.43` has `/Volumes/TRANSCEND`, about `3.6Ti` total and
|
| 525 |
+
`2.0Ti` available at inventory time. Use as source for corpus/dictionary
|
| 526 |
+
resources unless instructed otherwise.
|
| 527 |
+
- `cdjk@100.117.21.47` root was full at inspection time:
|
| 528 |
+
|
| 529 |
+
```text
|
| 530 |
+
/ 915G 865G 3.7G 100%
|
| 531 |
+
/mnt/tts_cache_nvme 234G 150G 72G 68%
|
| 532 |
+
```
|
| 533 |
+
|
| 534 |
+
Do not dump large generated data to a nearly-full root filesystem.
|
| 535 |
+
|
| 536 |
+
## Source Files To Inspect Before Editing
|
| 537 |
+
|
| 538 |
+
For Gorkhapatra flow:
|
| 539 |
+
|
| 540 |
+
```text
|
| 541 |
+
src/ocrtech/source_audit.py
|
| 542 |
+
src/ocrtech/cli.py
|
| 543 |
+
src/ocrtech/limbu_legacy.py
|
| 544 |
+
tests/test_source_audit.py
|
| 545 |
+
tests/test_limbu_legacy.py
|
| 546 |
+
```
|
| 547 |
+
|
| 548 |
+
For synthesis:
|
| 549 |
+
|
| 550 |
+
```text
|
| 551 |
+
src/ocrtech/synthesis_resources.py
|
| 552 |
+
tests/test_synthesis_resources.py
|
| 553 |
+
configs/cdjk-language-corpus-migration-v1.json
|
| 554 |
+
```
|
| 555 |
+
|
| 556 |
+
For fonts/language registry:
|
| 557 |
+
|
| 558 |
+
```text
|
| 559 |
+
src/ocrtech/font_assets.py
|
| 560 |
+
src/ocrtech/language_registry.py
|
| 561 |
+
configs/namdhinggo-font-assets-v1.json
|
| 562 |
+
configs/nepal-font-assets-v1.json
|
| 563 |
+
languages/nepal-v1.json
|
| 564 |
+
```
|
| 565 |
+
|
| 566 |
+
For training:
|
| 567 |
+
|
| 568 |
+
```text
|
| 569 |
+
src/ocrtech/training.py
|
| 570 |
+
src/ocrtech/models.py
|
| 571 |
+
docs/MODELS.md
|
| 572 |
+
docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md
|
| 573 |
+
```
|
| 574 |
+
|
| 575 |
+
## Known Pitfalls
|
| 576 |
+
|
| 577 |
+
- The user is frustrated by fake OCR candidates and manual busywork. Do not ask
|
| 578 |
+
them to validate bad OCR text when PDF-native text extraction can provide the
|
| 579 |
+
answer.
|
| 580 |
+
- For Limbu, PDF text was extractable but legacy encoded. The correct path was
|
| 581 |
+
PDF-native text -> Namdhinggo legacy conversion -> Unicode display.
|
| 582 |
+
- Dashboard bbox clicks must not reset scroll.
|
| 583 |
+
- Do not treat Devanagari-transcribed language data as native script OCR.
|
| 584 |
+
- Do not render mixed-script rows with a single script-specific font.
|
| 585 |
+
- Do not collapse mixed schemas into one converter command by dropping
|
| 586 |
+
validation. Split sources by schema instead.
|
| 587 |
+
- Do not publish synthetic-only claims.
|
| 588 |
+
|
| 589 |
+
## Recommended Next Commit
|
| 590 |
+
|
| 591 |
+
Implement the reusable validation generator and tests:
|
| 592 |
+
|
| 593 |
+
1. Refactor current generated dashboard logic into source code.
|
| 594 |
+
2. Add a CLI command that accepts a manifest/PDF-text extraction dir and emits
|
| 595 |
+
dashboard + review CSV/JSONL.
|
| 596 |
+
3. Preserve current Limbu dashboard features:
|
| 597 |
+
- full page image
|
| 598 |
+
- bbox overlay
|
| 599 |
+
- PDF-rendered span crop
|
| 600 |
+
- converted Unicode text
|
| 601 |
+
- raw legacy text
|
| 602 |
+
- font metadata
|
| 603 |
+
- stable scroll behavior
|
| 604 |
+
4. Add fixture tests for dashboard generation and annotation schema.
|
| 605 |
+
5. Document command in `docs/GORKHAPATRA_LANGUAGE_PAGE_ALIGNMENT.md`.
|
| 606 |
+
6. Run focused tests:
|
| 607 |
+
|
| 608 |
+
```bash
|
| 609 |
+
uv run pytest tests/test_source_audit.py tests/test_limbu_legacy.py -q
|
| 610 |
+
git diff --check
|
| 611 |
+
```
|
| 612 |
+
|
| 613 |
+
Then commit with GPG signing disabled and push.
|
ocr-tech-mps-integration/docs/CODEX_HANDOFF.md
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
ocr-tech-mps-integration/docs/DATASETS.md
ADDED
|
@@ -0,0 +1,381 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Dataset Workflow
|
| 2 |
+
|
| 3 |
+
The dataset workflow is deliberately strict. If train/eval contamination gets in here, every later validation result is suspect.
|
| 4 |
+
|
| 5 |
+
## Prepare
|
| 6 |
+
|
| 7 |
+
Convert raw labels to JSONL:
|
| 8 |
+
|
| 9 |
+
```bash
|
| 10 |
+
ocrtech prepare-data labels.csv \
|
| 11 |
+
--out data/manifests \
|
| 12 |
+
--dataset-name nepali-lines \
|
| 13 |
+
--split all \
|
| 14 |
+
--image-field image_path \
|
| 15 |
+
--text-field text \
|
| 16 |
+
--strict-chars
|
| 17 |
+
```
|
| 18 |
+
|
| 19 |
+
If the source is noisy and you need an explicit reject list instead of failing at the first bad row:
|
| 20 |
+
|
| 21 |
+
```bash
|
| 22 |
+
ocrtech prepare-data labels.csv \
|
| 23 |
+
--out data/manifests \
|
| 24 |
+
--dataset-name nepali-lines \
|
| 25 |
+
--split all \
|
| 26 |
+
--image-field image_path \
|
| 27 |
+
--text-field text \
|
| 28 |
+
--skip-invalid
|
| 29 |
+
```
|
| 30 |
+
|
| 31 |
+
That writes:
|
| 32 |
+
|
| 33 |
+
- `data/manifests/nepali-lines-all.jsonl`
|
| 34 |
+
- `data/manifests/nepali-lines-all-prepare.json`
|
| 35 |
+
- `data/manifests/nepali-lines-all-prepare.md`
|
| 36 |
+
- `data/manifests/nepali-lines-all-rejects.jsonl` when any rows are skipped
|
| 37 |
+
|
| 38 |
+
Verified Hugging Face sources can be converted directly:
|
| 39 |
+
|
| 40 |
+
```bash
|
| 41 |
+
ocrtech prepare-data gauravgiri/nepali-ocr-dataset \
|
| 42 |
+
--hf \
|
| 43 |
+
--split train \
|
| 44 |
+
--limit 100 \
|
| 45 |
+
--skip-invalid \
|
| 46 |
+
--out data/manifests
|
| 47 |
+
|
| 48 |
+
ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines \
|
| 49 |
+
--hf \
|
| 50 |
+
--split train \
|
| 51 |
+
--limit 100 \
|
| 52 |
+
--out data/manifests
|
| 53 |
+
|
| 54 |
+
ocrtech prepare-data nvidia/OCR-Synthetic-Multilingual-v1 \
|
| 55 |
+
--hf \
|
| 56 |
+
--hf-subset en \
|
| 57 |
+
--split test \
|
| 58 |
+
--limit 1000 \
|
| 59 |
+
--out data/manifests-english-test
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
Before those conversion commands, record a schema inspection so the training
|
| 63 |
+
corpus has evidence that fields were verified from the actual source:
|
| 64 |
+
|
| 65 |
+
```bash
|
| 66 |
+
ocrtech inspect-hf-dataset gauravgiri/nepali-ocr-dataset \
|
| 67 |
+
--split train \
|
| 68 |
+
--limit 3 \
|
| 69 |
+
--out outputs/hf-inspect-training-v1
|
| 70 |
+
|
| 71 |
+
ocrtech inspect-hf-dataset prashant0919/nepali-synthetic-ocr-lines \
|
| 72 |
+
--split train \
|
| 73 |
+
--limit 3 \
|
| 74 |
+
--out outputs/hf-inspect-training-v1
|
| 75 |
+
|
| 76 |
+
ocrtech inspect-hf-dataset nvidia/OCR-Synthetic-Multilingual-v1 \
|
| 77 |
+
--hf-subset en \
|
| 78 |
+
--split test \
|
| 79 |
+
--limit 3 \
|
| 80 |
+
--out outputs/hf-inspect-training-v1
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
Verified on `k2` on `2026-06-15`:
|
| 84 |
+
|
| 85 |
+
| dataset | split/subset | mode | inferred image | inferred text | result |
|
| 86 |
+
| --- | --- | --- | --- | --- | --- |
|
| 87 |
+
| `gauravgiri/nepali-ocr-dataset` | `train` | `datasets.load_dataset` | `image` | `text` | pass |
|
| 88 |
+
| `prashant0919/nepali-synthetic-ocr-lines` | `train` | HF repo `metadata.jsonl` | `file_name` | `text` | pass |
|
| 89 |
+
| `nvidia/OCR-Synthetic-Multilingual-v1` | `test/en` | HDF5 `line_bboxes` | `images + annotations.line_bboxes[].bbox` | `annotations.line_bboxes[].text` | pass |
|
| 90 |
+
|
| 91 |
+
Artifacts:
|
| 92 |
+
|
| 93 |
+
- `outputs/hf-inspect-training-v1/gauravgiri__nepali-ocr-dataset-train-inspect.json`
|
| 94 |
+
- `outputs/hf-inspect-training-v1/prashant0919__nepali-synthetic-ocr-lines-train-inspect.json`
|
| 95 |
+
- `outputs/hf-inspect-training-v1/nvidia__OCR-Synthetic-Multilingual-v1__en-test-inspect.json`
|
| 96 |
+
|
| 97 |
+
## Public Benchmarks
|
| 98 |
+
|
| 99 |
+
Public document parsing benchmarks are tracked separately from training data.
|
| 100 |
+
Do not mix them into recognizer/corrector training manifests. They are final
|
| 101 |
+
claim evidence.
|
| 102 |
+
|
| 103 |
+
List the known registry:
|
| 104 |
+
|
| 105 |
+
```bash
|
| 106 |
+
ocrtech list-public-benchmarks
|
| 107 |
+
```
|
| 108 |
+
|
| 109 |
+
Current target order:
|
| 110 |
+
|
| 111 |
+
1. `opendatalab/OmniDocBench` for broad document parsing.
|
| 112 |
+
2. `PaddlePaddle/Real5-OmniDocBench` for real-world physical distortions.
|
| 113 |
+
3. `allenai/olmOCR-bench` for PDF-to-Markdown/unit-test style evaluation.
|
| 114 |
+
|
| 115 |
+
Prepare the verified OmniDocBench schema:
|
| 116 |
+
|
| 117 |
+
```bash
|
| 118 |
+
ocrtech prepare-public-benchmark opendatalab/OmniDocBench \
|
| 119 |
+
--out data/public/omnidocbench \
|
| 120 |
+
--limit 100 \
|
| 121 |
+
--language english \
|
| 122 |
+
--require-tables
|
| 123 |
+
```
|
| 124 |
+
|
| 125 |
+
This converter is schema-bound. It expects the verified Hugging Face layout:
|
| 126 |
+
`OmniDocBench.json` as a list of rows, each row with `page_info.image_path`,
|
| 127 |
+
`page_info.page_attribute`, and `layout_dets`. Table references come from
|
| 128 |
+
`layout_dets[].html`; text references come from ordered `layout_dets[].text`
|
| 129 |
+
plus table HTML text. Images are resolved under `images/`.
|
| 130 |
+
|
| 131 |
+
Before benchmarking a public pack, audit its manifest:
|
| 132 |
+
|
| 133 |
+
```bash
|
| 134 |
+
ocrtech audit-public-benchmark data/public/omnidocbench/manifests/omnidocbench-eval.jsonl \
|
| 135 |
+
--benchmark opendatalab/OmniDocBench \
|
| 136 |
+
--out outputs/public-omnidocbench-audit
|
| 137 |
+
```
|
| 138 |
+
|
| 139 |
+
Each public manifest row must carry provenance:
|
| 140 |
+
|
| 141 |
+
- `metadata.public_benchmark = true`
|
| 142 |
+
- `metadata.benchmark_name`
|
| 143 |
+
- `metadata.source_url`
|
| 144 |
+
- `metadata.reference_path`
|
| 145 |
+
- `metadata.slices` containing `public`
|
| 146 |
+
|
| 147 |
+
This is intentionally stricter than `prepare-data`. The agent must verify the
|
| 148 |
+
actual downloaded dataset schema before writing a converter for any public
|
| 149 |
+
benchmark. No schema guessing.
|
| 150 |
+
|
| 151 |
+
Current claim-sized public OmniDocBench table pack on `k2`:
|
| 152 |
+
|
| 153 |
+
- manifest: `data/public/omnidocbench-table100/manifests/omnidocbench-eval.jsonl`
|
| 154 |
+
- prepare report: `data/public/omnidocbench-table100/public-benchmark-prepare.json`
|
| 155 |
+
- audit: `outputs/public-omnidocbench-table100-audit/public-benchmark-audit.json`
|
| 156 |
+
- audit status: passed
|
| 157 |
+
- samples: `100`
|
| 158 |
+
- slices: `public=100`, `document_parsing=100`, `english=100`, `table=100`,
|
| 159 |
+
`table_hard=32`, `figure=22`
|
| 160 |
+
|
| 161 |
+
This is the first public OmniDocBench manifest in the repo that can satisfy the
|
| 162 |
+
configured `100`-page public document gate and the `40`-page public table gate.
|
| 163 |
+
|
| 164 |
+
For the current Nepali starter corpus, merge verified sources with the built-in command instead of shell concatenation:
|
| 165 |
+
|
| 166 |
+
```bash
|
| 167 |
+
ocrtech merge-manifests \
|
| 168 |
+
data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl \
|
| 169 |
+
data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl \
|
| 170 |
+
--out data/manifests/nepali-mixed-all.jsonl
|
| 171 |
+
```
|
| 172 |
+
|
| 173 |
+
Do not pretend all HF datasets look the same. This repo only has verified adapters for the schemas we checked:
|
| 174 |
+
|
| 175 |
+
- `gauravgiri/nepali-ocr-dataset`: `image`, `text`
|
| 176 |
+
- `prashant0919/nepali-synthetic-ocr-lines`: `train/metadata.jsonl` plus image files
|
| 177 |
+
- `cfilt/RoundTripOCR-nepali`: `ocr`, `correct`, `font`
|
| 178 |
+
- `nvidia/OCR-Synthetic-Multilingual-v1`: HDF5 shards with `line_bboxes`
|
| 179 |
+
|
| 180 |
+
Known adapters also stamp verified slice metadata so later audits and experiments do not depend on hand-added tags:
|
| 181 |
+
|
| 182 |
+
- `gauravgiri/nepali-ocr-dataset`: `nepali`, `real`
|
| 183 |
+
- `prashant0919/nepali-synthetic-ocr-lines`: `nepali`, `synthetic`
|
| 184 |
+
- `nvidia/OCR-Synthetic-Multilingual-v1 --hf-subset en`: `english`, `synthetic`, `line_crop`, `en`
|
| 185 |
+
|
| 186 |
+
Verified on `k2` on June 14, 2026:
|
| 187 |
+
|
| 188 |
+
- `gauravgiri/nepali-ocr-dataset --limit 100 --skip-invalid`: `97` prepared, `3` rejected due to empty `text`
|
| 189 |
+
- `prashant0919/nepali-synthetic-ocr-lines --limit 100`: `100` prepared, `0` rejected
|
| 190 |
+
- merged starter corpus: `197` samples, no merge warnings
|
| 191 |
+
- deterministic split: `167` train / `30` eval, leakage check passed
|
| 192 |
+
|
| 193 |
+
Each manifest row contains:
|
| 194 |
+
|
| 195 |
+
- `sample_id`
|
| 196 |
+
- `dataset`
|
| 197 |
+
- `split`
|
| 198 |
+
- `image_path`
|
| 199 |
+
- `text`
|
| 200 |
+
- `sha256`
|
| 201 |
+
- `metadata.text_sha256`
|
| 202 |
+
- `metadata.sample_sha256`
|
| 203 |
+
|
| 204 |
+
## Audit
|
| 205 |
+
|
| 206 |
+
Before splitting or training:
|
| 207 |
+
|
| 208 |
+
```bash
|
| 209 |
+
ocrtech audit-manifest data/manifests/nepali-lines-all.jsonl \
|
| 210 |
+
--out outputs/manifest-audit \
|
| 211 |
+
--require-slice nepali \
|
| 212 |
+
--require-slice english \
|
| 213 |
+
--strict-chars
|
| 214 |
+
```
|
| 215 |
+
|
| 216 |
+
The audit checks:
|
| 217 |
+
|
| 218 |
+
- missing input files
|
| 219 |
+
- image SHA-256 mismatches
|
| 220 |
+
- duplicate sample IDs
|
| 221 |
+
- duplicate paths and hashes
|
| 222 |
+
- slice counts
|
| 223 |
+
- script counts
|
| 224 |
+
- unsupported characters when requested
|
| 225 |
+
|
| 226 |
+
## Split
|
| 227 |
+
|
| 228 |
+
Create train/eval manifests with deterministic grouping:
|
| 229 |
+
|
| 230 |
+
```bash
|
| 231 |
+
ocrtech split-manifest data/manifests/nepali-lines-all.jsonl \
|
| 232 |
+
--out data/splits \
|
| 233 |
+
--eval-ratio 0.15 \
|
| 234 |
+
--seed 13 \
|
| 235 |
+
--stratify-by slices
|
| 236 |
+
```
|
| 237 |
+
|
| 238 |
+
The splitter groups entries by `metadata.sample_sha256` before assigning splits by default. If the same image+text sample appears twice, it stays in one split.
|
| 239 |
+
|
| 240 |
+
For rendered packs with multiple degraded variants of the same underlying text, group by `text_sha256` instead. This keeps clean, blur, scan, low-contrast, and skew variants of the same target text on the same side of the train/eval boundary:
|
| 241 |
+
|
| 242 |
+
```bash
|
| 243 |
+
ocrtech split-manifest data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \
|
| 244 |
+
--out data/splits-table-figure-dev-v1 \
|
| 245 |
+
--eval-ratio 0.2 \
|
| 246 |
+
--seed 17 \
|
| 247 |
+
--stratify-by document_type \
|
| 248 |
+
--group-by text_sha256
|
| 249 |
+
```
|
| 250 |
+
|
| 251 |
+
Supported grouping keys are:
|
| 252 |
+
|
| 253 |
+
- `sample_signature`
|
| 254 |
+
- `text_sha256`
|
| 255 |
+
- `image_sha256`
|
| 256 |
+
- `image_path`
|
| 257 |
+
- `sample_id`
|
| 258 |
+
|
| 259 |
+
Outputs:
|
| 260 |
+
|
| 261 |
+
```text
|
| 262 |
+
data/splits/
|
| 263 |
+
train.jsonl
|
| 264 |
+
eval.jsonl
|
| 265 |
+
split-summary.json
|
| 266 |
+
split-summary.md
|
| 267 |
+
```
|
| 268 |
+
|
| 269 |
+
The split summary records the grouping key. For the table/figure development pack, the verified `k2` split produced:
|
| 270 |
+
|
| 271 |
+
- `60` train samples
|
| 272 |
+
- `20` eval samples
|
| 273 |
+
- `16` text groups
|
| 274 |
+
- `group_by = text_sha256`
|
| 275 |
+
- leakage check passed
|
| 276 |
+
|
| 277 |
+
## Rebalance
|
| 278 |
+
|
| 279 |
+
Use `rebalance-manifest` only on the training split when a slice is obviously underrepresented and you want a reproducible upsampling step instead of an undocumented training-time sampler choice:
|
| 280 |
+
|
| 281 |
+
```bash
|
| 282 |
+
ocrtech rebalance-manifest data/splits/train.jsonl \
|
| 283 |
+
--out data/splits/train-rebalanced.jsonl \
|
| 284 |
+
--slice english \
|
| 285 |
+
--slice nepali \
|
| 286 |
+
--seed 13
|
| 287 |
+
```
|
| 288 |
+
|
| 289 |
+
Outputs:
|
| 290 |
+
|
| 291 |
+
```text
|
| 292 |
+
data/splits/
|
| 293 |
+
train-rebalanced.jsonl
|
| 294 |
+
train-rebalanced-rebalance.json
|
| 295 |
+
train-rebalanced-rebalance.md
|
| 296 |
+
```
|
| 297 |
+
|
| 298 |
+
This command duplicates rows with explicit provenance in metadata:
|
| 299 |
+
|
| 300 |
+
- `rebalance_duplicate`
|
| 301 |
+
- `rebalance_source_sample_id`
|
| 302 |
+
- `rebalance_slice`
|
| 303 |
+
|
| 304 |
+
Do not rebalance the unsplit combined manifest. That would duplicate samples before train/eval assignment and corrupt the benchmark.
|
| 305 |
+
|
| 306 |
+
## Train
|
| 307 |
+
|
| 308 |
+
Use only the split outputs:
|
| 309 |
+
|
| 310 |
+
```bash
|
| 311 |
+
ocrtech train-recognizer \
|
| 312 |
+
--train-manifest data/splits/train.jsonl \
|
| 313 |
+
--eval-manifest data/splits/eval.jsonl \
|
| 314 |
+
--out runs/recognizer
|
| 315 |
+
```
|
| 316 |
+
|
| 317 |
+
For generated document packs, do not train a recognizer on whole rendered pages unless the model is explicitly document-level. The current full-page HF branch failed badly. Extract line/cell/caption crops first, then split those crop rows with text-level grouping:
|
| 318 |
+
|
| 319 |
+
```bash
|
| 320 |
+
ocrtech extract-crops \
|
| 321 |
+
data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \
|
| 322 |
+
--out data/crops-table-figure-dev-v1 \
|
| 323 |
+
--crop-type line \
|
| 324 |
+
--crop-type table-cell \
|
| 325 |
+
--crop-type figure-caption
|
| 326 |
+
|
| 327 |
+
ocrtech split-manifest data/crops-table-figure-dev-v1/manifests/crops.jsonl \
|
| 328 |
+
--out data/splits-crops-table-figure-dev-v1 \
|
| 329 |
+
--eval-ratio 0.2 \
|
| 330 |
+
--seed 17 \
|
| 331 |
+
--stratify-by document_type \
|
| 332 |
+
--group-by text_sha256
|
| 333 |
+
```
|
| 334 |
+
|
| 335 |
+
The crop extractor only reads explicit renderer layout metadata from the reference JSON. If that metadata is absent, it fails instead of guessing bboxes.
|
| 336 |
+
|
| 337 |
+
Before training, build the recognizer corpus from verified manifests with
|
| 338 |
+
explicit bucket targets:
|
| 339 |
+
|
| 340 |
+
```bash
|
| 341 |
+
ocrtech build-recognizer-corpus \
|
| 342 |
+
data/manifests/english-lines.jsonl \
|
| 343 |
+
data/manifests/nepali-lines.jsonl \
|
| 344 |
+
data/splits-crops-table-figure-dev-v1/train.jsonl \
|
| 345 |
+
--out data/splits/recognizer-train-balanced.jsonl \
|
| 346 |
+
--target-latin-only 200 \
|
| 347 |
+
--target-devanagari-only 200 \
|
| 348 |
+
--target-mixed 100 \
|
| 349 |
+
--target-real 100 \
|
| 350 |
+
--target-synthetic 100
|
| 351 |
+
```
|
| 352 |
+
|
| 353 |
+
This command upsamples existing examples only. If a requested bucket is missing
|
| 354 |
+
from the inputs, it fails; add source data instead of weakening the audit gate.
|
| 355 |
+
|
| 356 |
+
For OCR correction from the verified CFILT dataset:
|
| 357 |
+
|
| 358 |
+
```bash
|
| 359 |
+
ocrtech train-corrector \
|
| 360 |
+
--hf-dataset cfilt/RoundTripOCR-nepali \
|
| 361 |
+
--hf-split train \
|
| 362 |
+
--limit 1000 \
|
| 363 |
+
--out runs/corrector-cfilt
|
| 364 |
+
```
|
| 365 |
+
|
| 366 |
+
Do not point training at the combined manifest.
|
| 367 |
+
|
| 368 |
+
## Validation
|
| 369 |
+
|
| 370 |
+
Pass the training manifests into claim validation:
|
| 371 |
+
|
| 372 |
+
```bash
|
| 373 |
+
ocrtech validate-claim \
|
| 374 |
+
--benchmark-report outputs/benchmark/report.json \
|
| 375 |
+
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
|
| 376 |
+
--train-manifest data/splits/train.jsonl \
|
| 377 |
+
--config validation.json \
|
| 378 |
+
--out outputs/validation
|
| 379 |
+
```
|
| 380 |
+
|
| 381 |
+
The validation step checks train/eval leakage again. That is intentional redundancy.
|
ocr-tech-mps-integration/docs/EVAL_HARNESS_GOAL.md
ADDED
|
@@ -0,0 +1,437 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Evaluation Harness Goal
|
| 2 |
+
|
| 3 |
+
Updated: 2026-06-20
|
| 4 |
+
|
| 5 |
+
## Current Status
|
| 6 |
+
|
| 7 |
+
This is a parked claim-safety design, not the active product work. The active
|
| 8 |
+
2026-06-23 objective is Limbu book OCR: real book pages in, editable Unicode
|
| 9 |
+
book text out, with line structure preserved and unresolved text surfaced for
|
| 10 |
+
human review. Do not add generic eval-harness features from this document unless
|
| 11 |
+
they directly unblock that OCR loop. For the current Limbu work, the useful
|
| 12 |
+
evaluation surface is deliberately thin: frozen real-book page acceptance checks,
|
| 13 |
+
artifact audits, page/line counts, missed/extra/blank-line rates, review-packet
|
| 14 |
+
coverage, and reviewed-only training-pack export validation.
|
| 15 |
+
|
| 16 |
+
## Goal
|
| 17 |
+
|
| 18 |
+
The goal is a claim-grade OCR evaluation harness for the Nepal/diaspora recognizer
|
| 19 |
+
suite that makes weak publication claims difficult by default, not merely inconvenient:
|
| 20 |
+
each language/script result must be scored only from a frozen, versioned eval pack whose
|
| 21 |
+
source images, labels, manifest rows, source-document/page/bbox provenance, split role,
|
| 22 |
+
domain, ground-truth trust evidence, claim eligibility, canonical pack-content digest,
|
| 23 |
+
and script/grapheme coverage are
|
| 24 |
+
auditable; each run must record the exact model, checkpoint, config, dictionary,
|
| 25 |
+
prediction file, normalization profile, command, timestamp, machine, pack-content digest,
|
| 26 |
+
and artifact hashes;
|
| 27 |
+
each metric must be recomputed from canonical predictions as corpus micro edit-distance
|
| 28 |
+
CER over both Unicode codepoints and UAX #29 extended grapheme clusters, with exact-line
|
| 29 |
+
accuracy, matched/unmatched rows, worst examples, confidence intervals, and paired
|
| 30 |
+
comparison tests when claims compare systems; recognizer-only oracle-crop scores must
|
| 31 |
+
remain separate from page-level OCR scores that include detection, crop lineage, reading
|
| 32 |
+
order, and alignment; and the harness must fail or hard-warn on missing images, duplicate
|
| 33 |
+
IDs, low match rate, train/dev/test leakage, checkpoint selection on test data, synthetic
|
| 34 |
+
or transliterated data presented as real evidence, machine-converted labels presented as
|
| 35 |
+
human-verified evidence, human GT without review/agreement evidence, eval packs too small
|
| 36 |
+
for the claimed conclusion, uncovered script inventories, and aggregate averages that
|
| 37 |
+
hide per-language, per-script, per-domain, or per-GT-status failures. A reviewer opening
|
| 38 |
+
one output directory should be able to decide what was scored, against which ground truth,
|
| 39 |
+
how trustworthy that ground truth is, whether the test set was frozen and unseen for
|
| 40 |
+
selection, what normalization and metric were used, how uncertain the number is, what the
|
| 41 |
+
failures look like, what script material was not covered, and whether the result is safe
|
| 42 |
+
for a paper claim.
|
| 43 |
+
|
| 44 |
+
## Requirements
|
| 45 |
+
|
| 46 |
+
- **Eval packs are versioned evidence objects.** Each pack must include `images/`,
|
| 47 |
+
a label file, a manifest JSONL, and a summary/datasheet containing source documents,
|
| 48 |
+
page/bbox provenance, language, script, domain, split role, GT source, GT trust
|
| 49 |
+
level, known leakage status, claim eligibility, and a `pack_content_sha256` over the
|
| 50 |
+
canonical row identity/text/provenance fields so pack drift is detectable across
|
| 51 |
+
normalization, scoring, audit, and comparison.
|
| 52 |
+
- **Metrics are canonical and micro-averaged.** The harness must compute
|
| 53 |
+
total-edit-distance / total-reference-length CER, not per-line macro averages.
|
| 54 |
+
It must report both codepoint CER and grapheme-cluster CER, plus exact-line accuracy,
|
| 55 |
+
matched/unmatched counts, prediction match rate, total reference units, and
|
| 56 |
+
worst-line examples.
|
| 57 |
+
- **Point estimates are not enough for claims.** Every claim-facing metric must include
|
| 58 |
+
confidence intervals computed by resampling the scoring units, and same-pack system
|
| 59 |
+
comparisons must include a paired test over per-row errors. The report must show the
|
| 60 |
+
effective sample size per language/script/domain and hard-warn when a pack is too
|
| 61 |
+
small for the claimed conclusion.
|
| 62 |
+
- **PaddleOCR training metrics are not claim metrics.** PaddleOCR `RecMetric` reports
|
| 63 |
+
exact-line `acc` and `norm_edit_dis` as one minus the average per-sample normalized
|
| 64 |
+
Levenshtein distance; this may be useful during training, but it is not the
|
| 65 |
+
corpus-level micro CER used for claims.
|
| 66 |
+
- **Recognition and page OCR are separate tasks.** Oracle-crop recognizer results are
|
| 67 |
+
valid recognizer evidence only. Page-level OCR claims require page images, detector
|
| 68 |
+
output, crop/reading-order/alignment logic, and a separate end-to-end score.
|
| 69 |
+
- **Unicode handling is explicit.** Text is normalized with a named profile, defaulting
|
| 70 |
+
to NFC with whitespace preserved. The audit must reject a stale or unsupported
|
| 71 |
+
profile that no longer matches the metric recomputation path. Metrics must also
|
| 72 |
+
report how many raw GT/reference and prediction strings changed under that profile,
|
| 73 |
+
and the audit must recompute those counts from canonical row artifacts so hidden
|
| 74 |
+
normalization effects cannot disappear into a single CER. Grapheme metrics use
|
| 75 |
+
Unicode extended grapheme clusters; any script-specific tailoring must be recorded.
|
| 76 |
+
- **Human verification is first-class metadata.** Machine-converted GT may be scored,
|
| 77 |
+
but the output must label it as machine-converted. Paper-grade human-reading claims
|
| 78 |
+
require either fully reviewed GT or a documented native-reader calibration sample.
|
| 79 |
+
Human-verified GT must record who reviewed it, how much was double-reviewed, and the
|
| 80 |
+
observed disagreement or correction rate; a trust label without review evidence is not
|
| 81 |
+
claim evidence.
|
| 82 |
+
- **Script coverage is evidence, not decoration.** Each eval pack must report the
|
| 83 |
+
codepoint, grapheme-cluster, digit, punctuation, conjunct/combining-mark, and
|
| 84 |
+
language-specific inventory coverage it exercises, plus gaps against the intended
|
| 85 |
+
script inventory. A low CER on a pack that omits common graphemes or conjuncts must not
|
| 86 |
+
be eligible for broad script support claims.
|
| 87 |
+
- **Reports are inspectable.** Each run should emit `eval-run.json`, `metrics.json`,
|
| 88 |
+
`metrics.md`, `predictions.tsv`, source-audited `unmatched.tsv`, `worst-lines.csv`,
|
| 89 |
+
`worst-pages.csv`, `sample-review.csv`, `sample-review.html`, `confusions.json`, `command.txt`, and a
|
| 90 |
+
human-readable `claim-safety.md` report.
|
| 91 |
+
Worst-example rows must carry the row identity, hashes, language/script/domain/GT/split,
|
| 92 |
+
claim eligibility, source-document/page/bbox provenance, and reference-artifact trust
|
| 93 |
+
fields needed to inspect a failure without reverse-engineering the rest of the run.
|
| 94 |
+
- **Reporting is per language/script/domain.** Shared recognizers, such as the
|
| 95 |
+
Devanagari model, must still report Majhi, Magar, Sunuwar-Devanagari, Magahi,
|
| 96 |
+
Bhojpuri, Thami, Wambule, and Darai separately. Native-script targets such as
|
| 97 |
+
Limbu, Sunuwar/Jenticha, Kirat Rai, Lepcha, Tirhuta, Gurung Khema, and Newa must
|
| 98 |
+
stay separate from fallback or synthetic-only results.
|
| 99 |
+
|
| 100 |
+
## Current Implementation Slice
|
| 101 |
+
|
| 102 |
+
- `tools/run_recognizer_eval.py` scores an existing recognizer prediction file
|
| 103 |
+
against a PaddleOCR-style label file. It records non-empty model identity plus config/checkpoint/dictionary,
|
| 104 |
+
SHA-256 identities for local checkpoint/config/dictionary artifacts and ties those
|
| 105 |
+
identities back to the visible run fields, prediction, label, manifest,
|
| 106 |
+
summary/datasheet, script inventory, train/dev manifest, command, machine platform/Python metadata,
|
| 107 |
+
normalization, eval-pack version, frozen-pack declaration, selection declaration,
|
| 108 |
+
per-row crop image SHA-256 identities and source-document/page/bbox provenance in
|
| 109 |
+
`predictions.tsv` and `worst-lines.csv`, deterministic row-
|
| 110 |
+
bootstrap 95% confidence intervals for codepoint CER, grapheme CER, and exact-line
|
| 111 |
+
accuracy, observed and missing required script codepoint/grapheme inventory coverage,
|
| 112 |
+
and claim-safety warnings, and emits a direct `claim-safety.md` report in
|
| 113 |
+
the score directory. A claim-safe recognizer run now requires those artifacts to be
|
| 114 |
+
readable local files whose hashes can be audited and at least 30 scored rows per
|
| 115 |
+
emitted language/script/domain/GT/split slice so tiny or pooled smoke packs cannot
|
| 116 |
+
pass as paper evidence. Declared manifest crop-image hashes must
|
| 117 |
+
match actual crop bytes; missing script inventory files or missing required
|
| 118 |
+
codepoints/graphemes are claim-safety blockers, and rows declared `human_verified`
|
| 119 |
+
must carry human review evidence such as reviewer or review-artifact metadata.
|
| 120 |
+
- `tools/audit_recognizer_eval_pack.py` audits a recognizer oracle-crop eval pack
|
| 121 |
+
before scoring. It verifies label/image/manifest/summary integrity, duplicate
|
| 122 |
+
labels, manifest identifiers, and manifest image paths/basenames, manifest text/hash consistency, source-document/
|
| 123 |
+
page/bbox coverage, language/script/domain/GT/split metadata, claim eligibility,
|
| 124 |
+
frozen-pack declaration, unseen-for-selection declaration, and minimum scoring-unit
|
| 125 |
+
counts overall and per language/script/domain/GT/split slice.
|
| 126 |
+
- `tools/normalize_recognizer_eval_pack.py` converts existing PaddleOCR-style
|
| 127 |
+
recognizer packs into a self-contained canonical pack with copied images, normalized
|
| 128 |
+
`eval.txt`, `manifests/recognizer-eval.jsonl`, and `summary.json`. It preserves legacy
|
| 129 |
+
provenance fields, fills only explicitly provided defaults, hashes copied images, and
|
| 130 |
+
fails on missing images, duplicate labels, duplicate manifest IDs or image
|
| 131 |
+
paths/basenames, or manifest/label text mismatches.
|
| 132 |
+
- `configs/recognizer-eval-pack-normalization-v1.json` records reproducible
|
| 133 |
+
normalization recipes for the current real recognizer packs: Newa, Sunuwar, Lepcha,
|
| 134 |
+
Limbu, Kirat Rai, and shared Devanagari. `tools/run_recognizer_pack_normalization.py`
|
| 135 |
+
validates those recipes, supports dry-runs, materializes selected packs through the
|
| 136 |
+
normalizer, writes per-pack `audit/recognizer-eval-pack-audit.{json,md}` artifacts,
|
| 137 |
+
and records pack audit results in `normalization-run.json`.
|
| 138 |
+
- `configs/eval-target-coverage-v1.json` records the intended language/script coverage
|
| 139 |
+
sets: the broader first-ten document OCR targets and the current real recognizer
|
| 140 |
+
packs. `tools/validate_eval_target_coverage.py` compares those targets against the
|
| 141 |
+
normalization and eval-run recipe registries so missing packs, missing eval recipes,
|
| 142 |
+
non-claim-ready targets, and shared-pack language/script mismatches that could hide
|
| 143 |
+
per-language failures are explicit gaps. Target `eval_status` values are constrained
|
| 144 |
+
to known states, and only `registered_real_pack` can become coverage-ready; partial,
|
| 145 |
+
shared, missing, or invented status labels cannot silently pass as claim support.
|
| 146 |
+
- `tools/run_page_ocr_eval.py` scores saved page-level OCR `document.json` outputs
|
| 147 |
+
against structured references. It emits an explicit `eval_mode=page_level_ocr`
|
| 148 |
+
artifact, separates page OCR from recognizer oracle-crop metrics, records model/run
|
| 149 |
+
provenance with local checkpoint/config/dictionary SHA-256 identities, records
|
| 150 |
+
scored source image and `document.json` SHA-256 identities plus the embedded
|
| 151 |
+
`document.source_path` in `page-metrics.csv`, rejects claim-safe runs whose
|
| 152 |
+
`document.source_path` does not match the eval-manifest input, records and audits
|
| 153 |
+
the page pack `pack_content_sha256`, recomputes page text micro CER over codepoints
|
| 154 |
+
and grapheme clusters,
|
| 155 |
+
audits `page-metrics.csv` row identity back to the eval manifest, resolved
|
| 156 |
+
`document.json`, scorer metadata defaults, row claim eligibility, and parsed
|
| 157 |
+
reference fields, reports
|
| 158 |
+
auditable per-language/script/domain/GT/split page slices recomputed from
|
| 159 |
+
`page-metrics.csv`, existing document metrics such as reading-order pair accuracy,
|
| 160 |
+
table scores, figure scores, reconstruction
|
| 161 |
+
metrics, page text exact accuracy, canonical page prediction/reference text
|
| 162 |
+
pairs, per-row reference path/hash/trust artifacts, source-document/page/bbox provenance columns, reference coverage,
|
| 163 |
+
missing document/reference rows, row-level source-document/page/bbox/
|
| 164 |
+
claim-eligibility and language/script/domain/GT metadata coverage, row-level
|
| 165 |
+
claim eligibility, line-bbox IoU matching diagnostics when
|
| 166 |
+
references provide explicit reading-order bboxes,
|
| 167 |
+
worst-page review CSV/HTML artifacts, deterministic page-bootstrap 95% confidence
|
| 168 |
+
intervals for page text CER, observed and missing required script codepoint/grapheme
|
| 169 |
+
inventory coverage from reference text, and train/dev/eval leakage checks with recorded
|
| 170 |
+
train-manifest hashes. Page GT declared as `machine_converted_spot_checked` requires
|
| 171 |
+
an eligible native-reader calibration artifact, matching the recognizer-side trust
|
| 172 |
+
gate. Page claim-safe runs require at least 30 scored pages per emitted
|
| 173 |
+
language/script/domain/GT/split slice, and page score
|
| 174 |
+
directories also emit `claim-safety.md` so they can be reviewed without a recipe
|
| 175 |
+
wrapper. Duplicate page manifest sample IDs or image paths and stale manifest image
|
| 176 |
+
hashes fail before scoring, and missing script inventory files or missing required
|
| 177 |
+
codepoints/graphemes are claim-safety blockers. Page rows declared `human_verified`
|
| 178 |
+
must also carry human review evidence rather than relying on a trust label alone.
|
| 179 |
+
- `tools/run_eval_output_command.py` materializes canonical eval outputs through a
|
| 180 |
+
recorded external command before scoring. In `recognizer` mode it writes one
|
| 181 |
+
prediction file from a label pack; in `page` mode it runs once per eval-manifest row
|
| 182 |
+
and requires each sample to produce `document.json`. It records the exact expanded
|
| 183 |
+
command(s), command template, invocation, timestamp, stdout/stderr log paths, output
|
| 184 |
+
hashes/sizes, per-page input image paths/hashes, model/checkpoint/config/dictionary metadata, machine details, and
|
| 185 |
+
failure details with machine/invocation and label or eval-manifest hashes when a
|
| 186 |
+
command does not produce the required artifact. When
|
| 187 |
+
`--model-config` is provided without
|
| 188 |
+
`--command`, it uses the repo's candidate engine defaults: recognizer mode calls
|
| 189 |
+
`tools/predict_recognizer_eval.py`, while page mode calls `ocrtech parse-doc`.
|
| 190 |
+
- `tools/predict_recognizer_eval.py` turns a candidate model card into a canonical
|
| 191 |
+
recognizer prediction file for oracle-crop eval packs. It reads the label file,
|
| 192 |
+
runs the audited `candidate` engine on each crop, writes `image<TAB>prediction<TAB>
|
| 193 |
+
confidence`, and records model-card, label, output, machine, and command hashes.
|
| 194 |
+
- `tools/prepare_native_spotcheck_bundle.py` creates native-reader review bundles from
|
| 195 |
+
recognizer labels, and `tools/apply_native_spotcheck_reviews.py` ingests completed
|
| 196 |
+
review CSVs into calibration artifacts. The calibration step validates statuses,
|
| 197 |
+
emits reviewed-label and manifest-override artifacts, measures machine-converted label
|
| 198 |
+
CER against corrected/native-reviewed text, requires reviewer identity on reviewed
|
| 199 |
+
rows, and records whether the spot-check is eligible evidence for treating a pack as
|
| 200 |
+
`machine_converted_spot_checked`. Scorers bind that calibration evidence to the
|
| 201 |
+
current run's language, script, domain, `machine_converted` source GT status, and
|
| 202 |
+
`machine_converted_spot_checked` recommendation. Bundle summaries record source file
|
| 203 |
+
hashes, machine metadata, timestamp, and review artifact paths.
|
| 204 |
+
- `tools/build_recognizer_leakage_manifest.py` turns recognizer train/dev label files
|
| 205 |
+
into JSONL leakage manifests with sample IDs, image hashes, text hashes, and combined
|
| 206 |
+
sample hashes. These manifests are the standard input for scorer-side train/dev/eval
|
| 207 |
+
leakage checks; text-only mode exists for corpora where image files are intentionally
|
| 208 |
+
unavailable. Page-derived crop manifests must also preserve `source_sample_id`, and
|
| 209 |
+
the page scorer/audit treat train `source_sample_id` overlap with eval page
|
| 210 |
+
`sample_id` as hard leakage.
|
| 211 |
+
- `docs/EVAL_HARNESS_RUNBOOK.md` gives the checked operational recipe for chaining
|
| 212 |
+
normalization, leakage-manifest generation, output materialization, recognizer scoring,
|
| 213 |
+
native-reader calibration, and page-level scoring. It documents the exact claim-safety
|
| 214 |
+
flags and the distinction between repo candidate model-card runs and external
|
| 215 |
+
PaddleOCR/HF command templates.
|
| 216 |
+
- `configs/eval-harness-recipes-v1.json` records concrete current recipe instances for
|
| 217 |
+
Newa, Sunuwar, Lepcha, Limbu, Kirat Rai, shared Devanagari, and packaged HF/Paddle
|
| 218 |
+
recognizer smoke paths. `tools/validate_eval_harness_recipes.py` verifies that
|
| 219 |
+
referenced local paths exist, runs the canonical recognizer eval-pack audit for each
|
| 220 |
+
path-complete recognizer pack, statically audits page eval manifests for source-image
|
| 221 |
+
hashes, page provenance, row claim eligibility, human-review evidence, and
|
| 222 |
+
claim-ready references plus minimum overall and per-slice scoring-unit counts, and
|
| 223 |
+
separates structural recipe validity from claim readiness
|
| 224 |
+
blockers, including missing, malformed, or recipe-contradicting `summary_file` metadata
|
| 225 |
+
for language/script/task/dataset/domain/GT/split/version/eligibility,
|
| 226 |
+
missing or malformed `script_inventory_file`, incomplete row provenance,
|
| 227 |
+
pack-audit warnings or issues, missing, malformed, ineligible, or mismatched recipe-level
|
| 228 |
+
native spot-check calibration when default or audited row GT is `machine_converted_spot_checked`,
|
| 229 |
+
scorer-supported normalization profile declarations,
|
| 230 |
+
external command templates missing required input/output placeholders or containing
|
| 231 |
+
unsupported placeholders, unresolved external prediction command templates, commands still marked as
|
| 232 |
+
requiring operator resolution, unhashable checkpoint/config/dictionary artifacts,
|
| 233 |
+
malformed or empty `leakage_sources`, declared
|
| 234 |
+
checkpoint selection on the test pack, and declared known leakage.
|
| 235 |
+
- `tools/run_eval_recipe.py` is the recipe-level recognizer orchestrator. It validates
|
| 236 |
+
the selected recipe, builds or wires train/dev leakage manifests, materializes
|
| 237 |
+
predictions through the recorded runner, scores the canonical predictions, and writes
|
| 238 |
+
`recipe.json`, `recipe-validation.json`, `recipe-run.json`, root-level `audit.json`,
|
| 239 |
+
root-level `claim-safety.md`, materialization logs, leakage artifacts, and the scorer
|
| 240 |
+
output under one reviewable run directory. The root claim-safety report carries the
|
| 241 |
+
score-mode-specific recognizer or page facts from the recipe, score, and audit JSON
|
| 242 |
+
files, including script-inventory coverage, metadata contradictions, uncertainty
|
| 243 |
+
intervals, leakage status, and the score-mode review artifact list, so it can be
|
| 244 |
+
audited for stale reviewer-facing summaries, root `recipe.json` is tied back to the selected
|
| 245 |
+
recipe in the hashed registry file, root `recipe-validation.json` is tied back to
|
| 246 |
+
`recipe-run.json` and recomputed from the current validator, recipe-run
|
| 247 |
+
timestamp/machine/command provenance is checked, root
|
| 248 |
+
materialization/audit/claim-safety output pointers must point at the run directory's
|
| 249 |
+
own artifacts, root audit summaries must preserve current score-audit issue lists,
|
| 250 |
+
warnings, claim-safe status, and score-dir identity, root score summary paths and
|
| 251 |
+
hashes must match `score/metrics.json` and `score/eval-run.json`, root train-manifest
|
| 252 |
+
path/hash identities must match the score eval-run leakage inputs, the hashed recipe
|
| 253 |
+
registry file must sit under the declared recipe root, and the root
|
| 254 |
+
prediction/materialization paths, command template, output report, and materialized
|
| 255 |
+
model/checkpoint/config/dictionary metadata are tied back to the scored prediction
|
| 256 |
+
file hash and score eval-run metadata. Failed recipe attempts write
|
| 257 |
+
`recipe-run-error.json` with timestamp, machine, command, recipe validation,
|
| 258 |
+
registry hash, root, output directory, exception type, message, and traceback so a
|
| 259 |
+
failed run is still attributable.
|
| 260 |
+
- `tools/compare_recognizer_eval_runs.py` compares two completed recognizer oracle-crop
|
| 261 |
+
score directories on the same frozen eval pack. It pairs canonical `predictions.tsv`
|
| 262 |
+
rows by `sample_id`, verifies eval-pack/run identity including `pack_content_sha256`
|
| 263 |
+
and row GT identity, emits
|
| 264 |
+
`comparison.json`, `comparison.md`, `paired-rows.tsv`, and `command.txt`, reports candidate and
|
| 265 |
+
baseline micro CER over codepoints and graphemes, paired wins/losses/ties over
|
| 266 |
+
row-level CER, deterministic paired-row bootstrap confidence intervals for the CER
|
| 267 |
+
improvement, and an exact two-sided sign-test p-value. Its audit mode checks run
|
| 268 |
+
provenance, recomputes the comparison, and rejects stale JSON, Markdown, command,
|
| 269 |
+
or paired-row artifacts.
|
| 270 |
+
- `tools/compare_page_ocr_eval_runs.py` does the same for completed page-level OCR
|
| 271 |
+
score directories, using `page-eval-run.json`, `page-metrics.csv`, and
|
| 272 |
+
`text-pairs.tsv`. It verifies page eval-pack identity and page/reference text row
|
| 273 |
+
identity including `pack_content_sha256`, emits `comparison.json`,
|
| 274 |
+
`comparison.md`, `paired-pages.tsv`, and `command.txt`, reports
|
| 275 |
+
paired page text codepoint/grapheme CER improvements, deterministic paired-page
|
| 276 |
+
bootstrap intervals, and an exact two-sided sign-test p-value, and can audit
|
| 277 |
+
command/provenance plus the comparison artifacts back to the current score
|
| 278 |
+
directories.
|
| 279 |
+
- `tools/audit_eval_run.py` audits completed recognizer or page score directories, or a
|
| 280 |
+
recipe-run root with `score/`, for required artifacts, run metadata, metric arithmetic,
|
| 281 |
+
timestamp and command-file consistency, machine platform/Python provenance, recorded
|
| 282 |
+
input-file hash consistency, scorer-supported normalization profile consistency,
|
| 283 |
+
normalization-change summary recomputation from recognizer `predictions.tsv` or page
|
| 284 |
+
`text-pairs.tsv`,
|
| 285 |
+
model-artifact hash and run-field consistency,
|
| 286 |
+
recognizer `predictions.tsv` source-row recomputation from the recorded
|
| 287 |
+
label/prediction/manifest files, row-level
|
| 288 |
+
edit-distance and per-language/script/domain/GT/split slice recomputation,
|
| 289 |
+
recognizer crop-image and manifest-declared image-hash verification, recognizer source label/prediction
|
| 290 |
+
recomputation for label-line, matched-line, unmatched-label, unmatched-prediction,
|
| 291 |
+
match-rate metrics, deterministic bootstrap uncertainty intervals, and script
|
| 292 |
+
inventory recomputation from recognizer GT rows, recognizer
|
| 293 |
+
`unmatched.tsv` identity/hash/score recomputation from the source
|
| 294 |
+
label and prediction files, recognizer `confusions.json` recomputation from canonical
|
| 295 |
+
predictions, page `reference-artifacts.tsv` recomputation from scored reference
|
| 296 |
+
files, page `text-pairs.tsv` recomputation from scored document/reference text,
|
| 297 |
+
page `confusions.json` recomputation from those canonical text pairs,
|
| 298 |
+
page text CER row and total
|
| 299 |
+
recomputation from canonical page text pairs, page row provenance recomputation
|
| 300 |
+
from source manifests, scored source-image/
|
| 301 |
+
`document.json` hash verification, page bootstrap uncertainty interval recomputation,
|
| 302 |
+
script inventory recomputation from page reference text,
|
| 303 |
+
and page-metric identity recomputation from manifest/document/reference sources,
|
| 304 |
+
run-level language/script/domain/GT/split
|
| 305 |
+
list recomputation from canonical row artifacts, page
|
| 306 |
+
`line-alignment.csv` row-content and
|
| 307 |
+
detection/alignment recomputation from scored document/reference files, including
|
| 308 |
+
predicted/reference line IDs, bboxes, text, IoU, and per-line text CER, page
|
| 309 |
+
recognizer normalizer/defaulted-metadata count checks against the recorded
|
| 310 |
+
summary/datasheet,
|
| 311 |
+
direct-run summary/datasheet metadata consistency checks against run language,
|
| 312 |
+
script, domain, GT, split, eval-pack version, frozen/unseen, default
|
| 313 |
+
claim-eligibility, and task fields,
|
| 314 |
+
dictionary coverage recomputation from recognizer GT rows or page reference text,
|
| 315 |
+
with GT/reference codepoints outside the recorded dictionary treated as a
|
| 316 |
+
claim-safety blocker,
|
| 317 |
+
`missing.tsv` recomputation from the eval
|
| 318 |
+
manifest, document outputs, and reference files, slice totals, duplicate scored sample/image IDs
|
| 319 |
+
in completed output artifacts, ambiguous recognizer manifest image keys,
|
| 320 |
+
duplicate page manifest image paths, page manifest
|
| 321 |
+
image-hash mismatches, leakage-check presence and leakage-overlap recomputation
|
| 322 |
+
from train/eval sources, native spot-check calibration summary checks against the
|
| 323 |
+
recorded calibration artifact, metadata-contradiction recomputation from source
|
| 324 |
+
provenance signals, provenance/metadata coverage recomputation from source
|
| 325 |
+
manifests, page reference-coverage and manifest/scored/missing/claim-eligible count
|
| 326 |
+
recomputation from source artifacts, durable
|
| 327 |
+
claim-safety blockers
|
| 328 |
+
such as metadata contradictions, missing manifests or summary/datasheets, unrun
|
| 329 |
+
leakage checks, leakage overlaps, low match rate, unsafe domains, untrusted GT,
|
| 330 |
+
missing provenance, missing uncertainty evidence, too-small claim-facing sample size,
|
| 331 |
+
missing script inventory evidence, missing required script codepoints/graphemes,
|
| 332 |
+
zero-length recognizer GT rows or page text references, recognizer row-level claim
|
| 333 |
+
ineligibility, and incomplete page coverage,
|
| 334 |
+
warning/claim-safety
|
| 335 |
+
consistency, `metrics.md` consistency against `metrics.json`, recomputed
|
| 336 |
+
worst-example CSVs, reviewer CSV order/fields, and reviewer HTML contents, direct
|
| 337 |
+
score-directory claim reports whose core lines are checked against JSON run facts,
|
| 338 |
+
recipe-root claim reports whose recognizer/page score facts are checked against the
|
| 339 |
+
nested score directory, and materialization artifacts whose timestamp, machine platform/Python
|
| 340 |
+
and invocation metadata, JSON records, model/checkpoint/config/dictionary metadata,
|
| 341 |
+
static recipe-level dictionary coverage checks against recognizer GT or page
|
| 342 |
+
reference codepoints,
|
| 343 |
+
per-record commands/log paths, command-template file,
|
| 344 |
+
Markdown report, recognizer prediction path/hash or page document-output directory,
|
| 345 |
+
and output hashes are checked against the score
|
| 346 |
+
eval-run facts plus claim-report artifacts whose top-level report is also checked
|
| 347 |
+
against recipe and score JSON facts.
|
| 348 |
+
- The scorer computes corpus micro CER over NFC codepoints and Unicode extended
|
| 349 |
+
grapheme clusters, exact-line accuracy, matched/unmatched rows, per-language/script/
|
| 350 |
+
domain/GT/split slice metrics, worst lines/pages, review CSV/HTML, and recognizer/page
|
| 351 |
+
confusion counts.
|
| 352 |
+
- Claim-safety warnings now cover missing manifest or summary, missing eval-pack
|
| 353 |
+
version, missing script inventory files, missing required script codepoints or
|
| 354 |
+
graphemes, GT/reference codepoints outside the recorded dictionary, unhashable model
|
| 355 |
+
artifacts, zero-length recognizer GT rows or page text references, non-frozen or selection-used eval packs, low
|
| 356 |
+
prediction match rate, mixed languages/scripts/domains/GT statuses/splits, unsafe
|
| 357 |
+
domains, untrusted GT, metadata contradictions such as synthetic/transliterated
|
| 358 |
+
provenance declared as real material or machine-converted provenance declared as
|
| 359 |
+
`human_verified`, missing source-document/page/bbox/claim-eligibility provenance,
|
| 360 |
+
explicit row-level claim ineligibility, missing page row source-document/page/bbox/
|
| 361 |
+
claim-eligibility/language/script/domain/GT metadata, normalizer-filled metadata defaults from
|
| 362 |
+
canonical pack summaries, missing human-review evidence for `human_verified` rows, missing,
|
| 363 |
+
mismatched, or ineligible native spot-check calibration when
|
| 364 |
+
`machine_converted_spot_checked` is claimed, known leakage, unrun leakage checks, and
|
| 365 |
+
train/dev/eval overlaps from provided JSONL manifests.
|
| 366 |
+
- A Newa recognizer artifact was produced at
|
| 367 |
+
`outputs/evals/newa-prachalit/newa-native-rescore-2026-06-19-zenodo/`. It is
|
| 368 |
+
intentionally **not** claim-safe: GT is machine-converted, the pack is not declared
|
| 369 |
+
unseen for selection, page provenance and claim eligibility are missing, and no
|
| 370 |
+
train/dev leakage manifest was provided.
|
| 371 |
+
- The same Newa eval pack has a standalone audit at
|
| 372 |
+
`outputs/evals/newa-prachalit/newa-real-eval-pack-audit-2026-06-19/`. Basic pack
|
| 373 |
+
integrity passes (`3180/3180` labels matched to manifest rows and images), but
|
| 374 |
+
claim readiness fails because the pack is not declared unseen for selection, has
|
| 375 |
+
no page metadata, lacks row-level claim eligibility, and uses machine-converted GT.
|
| 376 |
+
|
| 377 |
+
## Remaining Gaps
|
| 378 |
+
|
| 379 |
+
- Resolve remaining remote-only checkpoint/package locations in
|
| 380 |
+
`configs/eval-harness-recipes-v1.json` and replace placeholder external commands with
|
| 381 |
+
runnable per-family commands. The recipe registry exists and validates local paths,
|
| 382 |
+
and `tools/run_eval_recipe.py` can execute resolved recognizer recipes, but several
|
| 383 |
+
language-specific PaddleOCR checkpoints are not packaged locally or hash-verifiable,
|
| 384 |
+
so the validator must keep them out of claim-ready status.
|
| 385 |
+
- Extend page-level evaluation with live detector output capture and crop lineage from
|
| 386 |
+
model runners; the current page harness scores saved structured documents and emits
|
| 387 |
+
per-line IoU/alignment rows when references provide explicit line bboxes, but it does
|
| 388 |
+
not yet run the detector/recognizer pipeline itself.
|
| 389 |
+
- Archive and review the materialized canonical eval packs under `outputs/eval-packs/`.
|
| 390 |
+
The six current real recognizer packs now exist with per-pack audits, but all six are
|
| 391 |
+
structurally non-claim-ready because of machine-converted GT, missing or defaulted
|
| 392 |
+
provenance/claim metadata, and for some packs selection-status blockers.
|
| 393 |
+
- Fill the target coverage gaps reported by `tools/validate_eval_target_coverage.py`.
|
| 394 |
+
The current coverage config validates, but is intentionally not coverage-ready:
|
| 395 |
+
English, Tamang, Magar, Wambule, Bantawa/Rai, Kulung, Yakkha, and Tharu/Rana Tharu
|
| 396 |
+
still lack registered real eval packs, and the current registered real-pack recipes
|
| 397 |
+
remain non-claim-ready because model commands, GT trust, provenance, calibration, or
|
| 398 |
+
row eligibility blockers remain unresolved. A target also remains non-coverage-ready
|
| 399 |
+
unless its `eval_status` is exactly `registered_real_pack`; use `partial_real_recognizer_pack`,
|
| 400 |
+
`shared_devanagari_pack`, or `missing_real_eval_pack` only when the resulting gap is
|
| 401 |
+
intended and visible.
|
| 402 |
+
- Generate and wire train/dev leakage manifests for each actual recognizer family and
|
| 403 |
+
pass them to scoring runs. The manifest builder exists, but each family still needs
|
| 404 |
+
its concrete train/dev label paths recorded in the run recipes, including page-origin
|
| 405 |
+
`source_sample_id` for crops selected from full-page eval material.
|
| 406 |
+
- Apply native-reader spot-check/calibration to the actual language eval packs, with
|
| 407 |
+
enough reviewed rows per script/domain to justify any `machine_converted_spot_checked`
|
| 408 |
+
upgrade. The harness now has artifacts and scorer wiring for this, but the reviewed
|
| 409 |
+
human evidence still has to be collected.
|
| 410 |
+
|
| 411 |
+
## Research Basis
|
| 412 |
+
|
| 413 |
+
- OCR-D states that OCR quality evaluation requires representative ground truth,
|
| 414 |
+
realistic images, manual transcriptions, localization of errors by document/page/
|
| 415 |
+
region, and repeatable workflows:
|
| 416 |
+
<https://ocr-d.de/en/spec/ocrd_eval.html>
|
| 417 |
+
- The OCR evaluation survey by Neudecker et al. describes CER as edit-distance errors
|
| 418 |
+
divided by reference character count and highlights that OCR evaluation depends on
|
| 419 |
+
alignment, reading order, layout, and tool-specific normalization choices:
|
| 420 |
+
<https://www.primaresearch.org/www/assets/papers/HIP21_CNeudecker_OcrEvalSurvey.pdf>
|
| 421 |
+
- Unicode UAX #29 defines extended grapheme clusters as the default programmatic
|
| 422 |
+
approximation of user-perceived characters, which is necessary for scripts where one
|
| 423 |
+
visible unit can span multiple codepoints:
|
| 424 |
+
<https://www.unicode.org/reports/tr29/>
|
| 425 |
+
- PaddleOCR's `RecMetric` implementation averages normalized Levenshtein distance per
|
| 426 |
+
sample and returns `norm_edit_dis`; this is not the same quantity as micro CER:
|
| 427 |
+
<https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/metrics/rec_metric.py>
|
| 428 |
+
- Datasheets for Datasets motivates documenting dataset motivation, composition,
|
| 429 |
+
collection, and recommended use, which maps directly onto eval-pack provenance:
|
| 430 |
+
<https://arxiv.org/abs/1803.09010>
|
| 431 |
+
- Model Cards for Model Reporting motivates model documentation with benchmarked
|
| 432 |
+
evaluation, intended use, limitations, and evaluation procedure:
|
| 433 |
+
<https://arxiv.org/abs/1810.03993>
|
| 434 |
+
- Recent OCR evaluation survey work warns that benchmark design can make historical
|
| 435 |
+
and marginalized documents invisible when evaluation emphasizes convenient modern
|
| 436 |
+
layouts rather than the actual target material:
|
| 437 |
+
<https://arxiv.org/html/2603.25761v1>
|
ocr-tech-mps-integration/docs/EVAL_HARNESS_RESEARCH_GOAL.md
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Research-Backed Eval Harness Goal
|
| 2 |
+
|
| 3 |
+
Updated: 2026-06-20
|
| 4 |
+
|
| 5 |
+
## Sources Read
|
| 6 |
+
|
| 7 |
+
- Web/paper search performed on 2026-06-20 before drafting this goal.
|
| 8 |
+
- OCR-D quality assurance guidance: representative ground truth, realistic images,
|
| 9 |
+
manual transcription, page/region localization, CER/WER definitions, reading-order
|
| 10 |
+
and layout caveats, IoU/object-detection evaluation, and NFC normalization guidance:
|
| 11 |
+
<https://ocr-d.de/en/spec/ocrd_eval.html>
|
| 12 |
+
- Neudecker et al., "A survey of OCR evaluation tools and metrics" (HIP 2021):
|
| 13 |
+
OCR evaluation is tool- and normalization-sensitive, and text accuracy depends on
|
| 14 |
+
alignment, reading order, layout, and the exact edit-distance metric:
|
| 15 |
+
<https://dl.acm.org/doi/10.1145/3476887.3476888>
|
| 16 |
+
- Santos, "OCR Evaluation Tools for the 21st Century" (ComputEL 2019): OCR evaluation
|
| 17 |
+
for modern and endangered-language work needs Unicode-capable character/word
|
| 18 |
+
reporting rather than ASCII-era assumptions:
|
| 19 |
+
<https://doi.org/10.33011/computel.v1i.345>
|
| 20 |
+
- Unicode Standard Annex #29: extended grapheme clusters are the default
|
| 21 |
+
programmatic approximation of user-perceived characters:
|
| 22 |
+
<https://www.unicode.org/reports/tr29/>
|
| 23 |
+
- PaddleOCR `RecMetric`: `norm_edit_dis` is one minus a per-sample average of
|
| 24 |
+
normalized Levenshtein distance, not corpus micro CER:
|
| 25 |
+
<https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/metrics/rec_metric.py>
|
| 26 |
+
- Leifert et al., "End-to-End Measure for Text Recognition" (ICDAR 2019): end-to-end
|
| 27 |
+
text recognition evaluation must account for recognition plus segmentation/alignment,
|
| 28 |
+
not just isolated text strings:
|
| 29 |
+
<https://doi.org/10.1109/ICDAR.2019.00-16>
|
| 30 |
+
- Beyene and Dancy, "A Survey of OCR Evaluation Methods and Metrics and the
|
| 31 |
+
Invisibility of Historical Documents" (arXiv 2026 preprint): modern OCR and document
|
| 32 |
+
understanding evaluations overfit to modern/institutional benchmark genres, often
|
| 33 |
+
miss historical/community archive layout failures, and can make marginalized
|
| 34 |
+
materials invisible even when headline CER/WER looks acceptable:
|
| 35 |
+
<https://arxiv.org/html/2603.25761v1>
|
| 36 |
+
- Springmann et al., "Automatic quality evaluation and (semi-) automatic improvement
|
| 37 |
+
of OCR models for historical printings": GT is scarce, manual lines can change
|
| 38 |
+
results materially, and confidence/proxy measures are not replacements for true GT:
|
| 39 |
+
<https://arxiv.org/abs/1606.05157>
|
| 40 |
+
- Gebru et al., "Datasheets for Datasets": datasets need explicit motivation,
|
| 41 |
+
composition, collection, processing, intended use, and limitations:
|
| 42 |
+
<https://arxiv.org/abs/1803.09010>
|
| 43 |
+
- Mitchell et al., "Model Cards for Model Reporting": model releases need benchmarked
|
| 44 |
+
performance, intended use, limitations, evaluation procedure, and per-condition
|
| 45 |
+
reporting:
|
| 46 |
+
<https://arxiv.org/abs/1810.03993>
|
| 47 |
+
|
| 48 |
+
## Dense Goal Paragraph
|
| 49 |
+
|
| 50 |
+
Build a claim-grade OCR evaluation harness for the Nepal/diaspora recognizer suite
|
| 51 |
+
whose default output makes an unsafe paper claim obvious before anyone can quote a
|
| 52 |
+
number: every supported target, including the shared Devanagari languages
|
| 53 |
+
Majhi, Magar, Sunuwar-Devanagari, Magahi, Bhojpuri, Thami, Wambule, and Darai plus
|
| 54 |
+
native-script targets such as Limbu/Sirijonga, Sunuwar/Jenticha, Newa/Prachalit,
|
| 55 |
+
Tirhuta, Lepcha, Kirat Rai, Gurung Khema, and any later Tamang, Bantawa/Rai, Kulung,
|
| 56 |
+
Yakkha, Tharu/Rana Tharu, or English scope, must be evaluated as its own
|
| 57 |
+
language/script/domain slice from a frozen, versioned, real-image eval pack with
|
| 58 |
+
image bytes, labels, manifest rows, source-document/page/bbox provenance, split role,
|
| 59 |
+
domain, ground-truth source and trust evidence, native-reader calibration when GT is
|
| 60 |
+
machine-converted, row claim eligibility, script inventory coverage, and leakage
|
| 61 |
+
declarations; every run must record exact model/checkpoint/config/dictionary or model
|
| 62 |
+
card, prediction artifact, normalization profile, Unicode and grapheme-segmentation
|
| 63 |
+
behavior, command, timestamp, machine, tool versions, and artifact hashes; every
|
| 64 |
+
reported metric must be recomputable from canonical row artifacts as corpus micro
|
| 65 |
+
edit-distance CER over both Unicode codepoints and UAX #29 extended grapheme clusters,
|
| 66 |
+
not PaddleOCR `norm_edit_dis` or any per-line macro surrogate, with exact-line
|
| 67 |
+
accuracy, matched and unmatched rows, missing-prediction accounting, confusion
|
| 68 |
+
summaries, worst examples, reviewer artifacts, deterministic confidence intervals,
|
| 69 |
+
and paired same-pack tests for model comparisons; recognizer oracle-crop results must
|
| 70 |
+
remain separate from page-level OCR results, where page claims additionally require
|
| 71 |
+
detection/crop lineage, reading-order and layout alignment, line-bbox/IoU or explicit
|
| 72 |
+
reference matching, table/figure/text reconstruction metrics where relevant,
|
| 73 |
+
historical/newspaper degradation and multi-column failure visibility, and visible
|
| 74 |
+
coverage of missing documents or references; the harness must fail or hard-warn on
|
| 75 |
+
missing or mutated files, duplicate IDs or image paths, stale hashes, unsupported or
|
| 76 |
+
asymmetric normalization, low match rate, unrun or positive train/dev/test leakage,
|
| 77 |
+
checkpoint selection on test data, synthetic or rendered data presented as real
|
| 78 |
+
evidence, transliterated or machine-converted labels presented as human-verified
|
| 79 |
+
evidence, human GT without review/agreement evidence, too-small or script-incomplete
|
| 80 |
+
packs, aggregate scores that hide per-language/script/domain/GT failures, modern-
|
| 81 |
+
benchmark averages that do not exercise the target community archives, and any
|
| 82 |
+
hand-edited report whose JSON/TSV/CSV artifacts no longer recompute; a reviewer
|
| 83 |
+
opening one output directory must be able to answer, without trusting prose, exactly
|
| 84 |
+
what was scored, against which GT, how trustworthy that GT is, whether the test set was
|
| 85 |
+
frozen and unseen for selection, which Unicode, script, layout, provenance, and archive
|
| 86 |
+
coverage assumptions moved the number, how uncertain the number is, what failed, whose
|
| 87 |
+
material is still unsupported, and whether the result is safe to cite in a paper.
|
| 88 |
+
|
| 89 |
+
## Non-Negotiable Requirements
|
| 90 |
+
|
| 91 |
+
- Real held-out eval packs are evidence objects, not folders of images. They need
|
| 92 |
+
immutable manifests, hashes, provenance, split status, GT trust, domain labels,
|
| 93 |
+
claim eligibility, script inventory coverage, and a datasheet-style summary.
|
| 94 |
+
- Recognizer and page OCR are different tasks. Oracle-crop CER may support a recognizer
|
| 95 |
+
claim only; page OCR claims need detector output, crop lineage, reading order,
|
| 96 |
+
layout alignment, and missing-page/reference accounting.
|
| 97 |
+
- Metrics must be recomputed from emitted row artifacts. Aggregate JSON is not a
|
| 98 |
+
source of truth unless `predictions.tsv`, `page-metrics.csv`, alignment rows, and
|
| 99 |
+
references recompute to the same values.
|
| 100 |
+
- Unicode is part of the metric contract. The harness must record normalization,
|
| 101 |
+
Unicode/grapheme implementation behavior, codepoint CER, grapheme CER, and missing
|
| 102 |
+
required script inventory items.
|
| 103 |
+
- Claim-facing reports need uncertainty and comparison discipline. Point CER alone is
|
| 104 |
+
not enough; small packs need hard warnings, and model-vs-model claims need paired
|
| 105 |
+
same-pack analysis.
|
| 106 |
+
- Ground truth quality must be visible. Machine conversion and transliteration are
|
| 107 |
+
scoreable, but they are not human-verified evidence unless a native-reader
|
| 108 |
+
calibration artifact supports that upgrade.
|
| 109 |
+
- The harness must be hostile to silent omissions. Missing predictions, duplicate
|
| 110 |
+
rows, leaked train/dev examples, stale hashes, skipped pages, unsupported
|
| 111 |
+
normalization, and stale Markdown should fail audit or block claim safety.
|
ocr-tech-mps-integration/docs/EVAL_HARNESS_RUNBOOK.md
ADDED
|
@@ -0,0 +1,803 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Eval Harness Runbook
|
| 2 |
+
|
| 3 |
+
Updated: 2026-06-20
|
| 4 |
+
|
| 5 |
+
This is the operational path for producing a claim-grade OCR eval directory. The
|
| 6 |
+
goal is not to get a number quickly; it is to make the number hard to misread.
|
| 7 |
+
Each run should leave enough artifacts for a reviewer to answer what was scored,
|
| 8 |
+
which model produced it, which ground truth was used, whether the pack was frozen
|
| 9 |
+
and unseen for selection, what leakage checks ran, and why the result is or is not
|
| 10 |
+
safe for a paper claim.
|
| 11 |
+
|
| 12 |
+
## Recipe-First Eval
|
| 13 |
+
|
| 14 |
+
Use this path when a registered recipe exists for either `recognizer_oracle_crop`
|
| 15 |
+
or `page_level_ocr`. Recipes keep validation, leakage checks, output
|
| 16 |
+
materialization, scoring, root reports, and audit metadata under one output
|
| 17 |
+
directory.
|
| 18 |
+
|
| 19 |
+
### Registry Validation
|
| 20 |
+
|
| 21 |
+
When a recipe exists in `configs/eval-harness-recipes-v1.json`, use it as the
|
| 22 |
+
default entry point. This keeps validation, leakage checks, prediction
|
| 23 |
+
materialization, and scoring under one output directory.
|
| 24 |
+
|
| 25 |
+
Before scoring legacy source packs, validate or run the normalization registry:
|
| 26 |
+
|
| 27 |
+
```bash
|
| 28 |
+
uv run python tools/run_recognizer_pack_normalization.py \
|
| 29 |
+
--config configs/recognizer-eval-pack-normalization-v1.json \
|
| 30 |
+
--root . \
|
| 31 |
+
--all \
|
| 32 |
+
--out outputs/eval-packs/normalization-runs/<run-id> \
|
| 33 |
+
--dry-run \
|
| 34 |
+
--require-valid
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
Drop `--dry-run` only when you intentionally want to materialize canonical packs
|
| 38 |
+
under `outputs/eval-packs/<pack-version>/`. A materialized pack gets its own
|
| 39 |
+
`audit/recognizer-eval-pack-audit.json` and `.md`; read those before wiring the
|
| 40 |
+
pack into a claim run.
|
| 41 |
+
|
| 42 |
+
Then validate intended language/script coverage against the normalization and
|
| 43 |
+
eval-run recipe registries:
|
| 44 |
+
|
| 45 |
+
```bash
|
| 46 |
+
uv run python tools/validate_eval_target_coverage.py \
|
| 47 |
+
--coverage-config configs/eval-target-coverage-v1.json \
|
| 48 |
+
--normalization-config configs/recognizer-eval-pack-normalization-v1.json \
|
| 49 |
+
--eval-recipes configs/eval-harness-recipes-v1.json \
|
| 50 |
+
--root . \
|
| 51 |
+
--out outputs/evals/target-coverage.json \
|
| 52 |
+
--require-valid
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
Do not add `--require-coverage-ready` until every target in the selected claim
|
| 56 |
+
scope has a normalized eval pack, runnable eval recipe, and no claim blockers.
|
| 57 |
+
If a target points at a shared aggregate pack, the validator must show matching
|
| 58 |
+
target language/script metadata or an explicit gap; do not use a shared pack
|
| 59 |
+
average as evidence for an individual language claim. Coverage target status is
|
| 60 |
+
also gated: `eval_status` must be one of the known states, and only
|
| 61 |
+
`registered_real_pack` is allowed to become coverage-ready. Use
|
| 62 |
+
`partial_real_recognizer_pack`, `shared_devanagari_pack`, or
|
| 63 |
+
`missing_real_eval_pack` when a target is deliberately not ready for claims.
|
| 64 |
+
For book OCR targets, set `requires_page_eval_recipe` and
|
| 65 |
+
`requires_book_candidate_evidence` on the target set or target. That makes coverage
|
| 66 |
+
depend on a page-level recipe with book-candidate evidence from the scoring/admission
|
| 67 |
+
lane, not just an oracle-crop recognizer score or a loose page replay. The validator
|
| 68 |
+
also reports whether the linked book candidate is operationally admitted and
|
| 69 |
+
claim-safe, so non-claim-safe Sirijonga and Devanagari book evidence show up as
|
| 70 |
+
separate gaps.
|
| 71 |
+
|
| 72 |
+
```bash
|
| 73 |
+
uv run python tools/validate_eval_harness_recipes.py \
|
| 74 |
+
--recipes configs/eval-harness-recipes-v1.json \
|
| 75 |
+
--root . \
|
| 76 |
+
--require-valid
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
Each recipe `eval_pack` must declare `normalization_profile:
|
| 80 |
+
NFC_preserve_whitespace`, matching the scorer/auditor recomputation contract, and a
|
| 81 |
+
readable, parseable `summary_file` and `script_inventory_file`. The recipe validator
|
| 82 |
+
checks the summary JSON object and inventory JSON shape before a recipe is considered
|
| 83 |
+
structurally valid, and a summary without dataset/version/eval-pack metadata blocks
|
| 84 |
+
claim readiness. Summary/datasheet declarations for language, script, dataset, task,
|
| 85 |
+
domain, GT status, split, pack version, frozen/unseen status, and default claim
|
| 86 |
+
eligibility must agree with the recipe metadata; contradictions block claim readiness.
|
| 87 |
+
Do not paper over a pack that needs another normalization policy; add
|
| 88 |
+
scorer support and audit coverage first, then change the recipe.
|
| 89 |
+
The script inventory is a JSON object with `required_codepoints` and
|
| 90 |
+
`required_graphemes` lists. Codepoints may be written as `U+XXXX` values or literal
|
| 91 |
+
single characters; graphemes are literal strings. The scorer records observed coverage
|
| 92 |
+
from GT/reference text, and claim-safe runs fail if any required item is missing.
|
| 93 |
+
When a runner dictionary is a readable local file, the recipe validator also compares
|
| 94 |
+
its non-whitespace codepoints to recognizer GT or page reference text; out-of-vocabulary
|
| 95 |
+
GT/reference codepoints block claim readiness before scoring.
|
| 96 |
+
If the recipe default or audited row metadata uses `machine_converted_spot_checked`
|
| 97 |
+
GT, the recipe must also declare a readable JSON `eval_pack.spotcheck_calibration_file`;
|
| 98 |
+
the recipe validator checks its eligibility/recommendation/source-status fields and
|
| 99 |
+
language/script/domain consistency, and recipe execution passes that artifact into the
|
| 100 |
+
scorer, where it is hashed and audited.
|
| 101 |
+
|
| 102 |
+
### Pack Content Digest Contract
|
| 103 |
+
|
| 104 |
+
Every claim-facing eval pack summary/datasheet must include `pack_content_sha256`.
|
| 105 |
+
This is not a cosmetic version string. It is the immutable content identity that lets
|
| 106 |
+
scoring, audit, and comparison prove that the score still belongs to the same source
|
| 107 |
+
pack.
|
| 108 |
+
|
| 109 |
+
For recognizer oracle-crop packs, the digest binds each canonical row's sample ID,
|
| 110 |
+
source image hash, NFC-normalized GT text hash, language, script, domain, GT status,
|
| 111 |
+
split, and claim eligibility. For page-level OCR packs, the digest binds each canonical
|
| 112 |
+
page row's sample ID, input image hash, reference artifact hash, reference text hash,
|
| 113 |
+
reference fields, reference trust metadata, source-document/page/bbox provenance,
|
| 114 |
+
language, script, domain, GT status, split, and claim eligibility.
|
| 115 |
+
|
| 116 |
+
Claim-safe scoring must fail when the summary digest is missing or stale. Completed
|
| 117 |
+
score directories must record the recomputed value in `metrics.json` and in the
|
| 118 |
+
mode-specific run file (`eval-run.json` for recognizer, `page-eval-run.json` for page
|
| 119 |
+
OCR). Audit must recompute the digest from current source artifacts and reject any
|
| 120 |
+
mismatch in the run JSON, metrics JSON, or summary/datasheet. Same-task comparison
|
| 121 |
+
tools must treat the digest as eval-pack identity, so a candidate and baseline cannot
|
| 122 |
+
be compared as claim-safe if their `pack_content_sha256` values differ.
|
| 123 |
+
|
| 124 |
+
When a test or fixture intentionally edits pack content before scoring, regenerate the
|
| 125 |
+
summary digest before running the scorer. When content changes after scoring, do not
|
| 126 |
+
refresh the digest; that is tampering or drift, and audit should fail.
|
| 127 |
+
|
| 128 |
+
For `page_level_ocr` recipes, `eval_pack.eval_manifest` is the canonical page
|
| 129 |
+
manifest and materialization writes per-page `document.json` files under the root
|
| 130 |
+
`documents/` directory. External page command templates must write either
|
| 131 |
+
`{document_json}` directly or write `document.json` below `{output_dir}`. The recipe
|
| 132 |
+
validator statically audits page manifests for readable source images and hashes,
|
| 133 |
+
source-document/page/bbox provenance, row claim eligibility, human-review evidence for
|
| 134 |
+
`human_verified` rows, parseable references, claim-ready reference metadata, and
|
| 135 |
+
minimum scoring-unit counts overall and per language/script/domain/GT/split slice; any
|
| 136 |
+
page audit issue or warning blocks claim readiness. For path-complete recognizer packs,
|
| 137 |
+
the recipe validator also runs
|
| 138 |
+
`tools/audit_recognizer_eval_pack.py`; any pack-audit issue or warning becomes a
|
| 139 |
+
claim-readiness blocker, including overall and language/script/domain/GT/split
|
| 140 |
+
sample-size warnings below the claim minimum. `leakage_sources` must also be
|
| 141 |
+
parseable and non-empty; page-level leakage manifests must satisfy the page manifest
|
| 142 |
+
schema, and recognizer label leakage sources must contain valid rows with existing
|
| 143 |
+
images. A recipe is not claim-ready merely because its label, manifest, summary,
|
| 144 |
+
script inventory, and image paths exist.
|
| 145 |
+
|
| 146 |
+
For Limbu, keep the pack counts straight. The canonical materialized recognizer pack
|
| 147 |
+
`outputs/eval-packs/limbu-gorkhapatra-real-v1/` has 1,829 labels and 1,829 manifest
|
| 148 |
+
rows, not 39. The number 39 belongs to an older operational page run's unresolved
|
| 149 |
+
review queue and must not be described as the available Limbu eval set. The current
|
| 150 |
+
Limbu recognizer pack is structurally auditable but not claim-ready: it is
|
| 151 |
+
machine-converted, has 0 claim-eligible rows, is not declared unseen for selection, and
|
| 152 |
+
is missing source-document/page/bbox provenance. The separate CDC Maths PDF text-layer
|
| 153 |
+
pack has 81 silver rendered-line rows, split 69 train / 12 eval; use it for recognizer
|
| 154 |
+
bootstrap/training experiments, not for claim-safe evaluation.
|
| 155 |
+
For `page_level_ocr` recipes with `ocr_output_kind=live_detector`, the recipe must
|
| 156 |
+
also declare `runner.output_audit` bound to `{output_dir}` so page scoring is paired
|
| 157 |
+
with an operational artifact audit. Limbu page recipes must run `audit-limbu-output`
|
| 158 |
+
and record strict flags including `--require-capture-prep`, prepared-image quality
|
| 159 |
+
thresholds (`--min-capture-prepared-width`, `--min-capture-prepared-height`,
|
| 160 |
+
`--min-capture-prepared-entropy`, `--min-capture-prepared-luminance-stddev`, and
|
| 161 |
+
`--min-capture-prepared-edge-stddev`), `--require-no-pending-review`,
|
| 162 |
+
`--require-no-dropped-image-lines`, output quality/volume gates
|
| 163 |
+
(`--min-line-count`, `--min-average-line-confidence`, and `--min-quality-score`), and
|
| 164 |
+
the script-specific `--require-script` value for their target, for example
|
| 165 |
+
`limbu_sirijonga` for Sirijonga-script Limbu or `devanagari_limbu` for
|
| 166 |
+
Devanagari-written Limbu. This keeps page scoring tied to the operational artifact
|
| 167 |
+
audit instead of allowing a claim-ready registry entry for an OCR bundle that would
|
| 168 |
+
fail provenance, capture-quality, output-volume, output-quality, review-resolution, or
|
| 169 |
+
complete-line-retention gates.
|
| 170 |
+
|
| 171 |
+
Then run the selected recipe:
|
| 172 |
+
|
| 173 |
+
```bash
|
| 174 |
+
uv run python tools/run_eval_recipe.py \
|
| 175 |
+
--recipes configs/eval-harness-recipes-v1.json \
|
| 176 |
+
--recipe-id <recipe-id> \
|
| 177 |
+
--root . \
|
| 178 |
+
--out outputs/evals/<run-id> \
|
| 179 |
+
--timeout-seconds 3600 \
|
| 180 |
+
--min-match-rate 0.99
|
| 181 |
+
```
|
| 182 |
+
|
| 183 |
+
Add `--require-claim-safe` only when the recipe validator reports no claim
|
| 184 |
+
blockers and the scorer is expected to emit no warnings. If the recipe uses an
|
| 185 |
+
external command template, replace the placeholder command in the recipe first;
|
| 186 |
+
unresolved templates are rejected by default. Recognizer external templates must include
|
| 187 |
+
`{label_file}`, `{data_dir}`, and `{prediction_file}`; page external templates must
|
| 188 |
+
include `{input_path}` plus `{document_json}` or `{output_dir}`. Unknown placeholders
|
| 189 |
+
are validation errors, so command templates must use only the mode-supported
|
| 190 |
+
placeholders documented by `tools/run_eval_output_command.py`. Recipes that declare
|
| 191 |
+
`runner.selected_checkpoint_on_test` or non-empty `known_leakage` remain structurally
|
| 192 |
+
valid but are blocked from claim readiness because the scorer will warn on those same
|
| 193 |
+
facts at runtime. Runner `checkpoint`, `config`, and `dictionary` values must point to
|
| 194 |
+
readable local files, not directories, because claim-safe scores record and audit their
|
| 195 |
+
SHA-256 hashes.
|
| 196 |
+
Recognizer and page eval-pack summaries must also carry `pack_content_sha256`, the
|
| 197 |
+
canonical digest over sample ID, input/reference hashes, normalized GT text or
|
| 198 |
+
structured-reference text, provenance, language, script, domain, GT status, split, and
|
| 199 |
+
claim eligibility. The scorers recompute this digest from the rows they actually
|
| 200 |
+
scored; claim-safe runs fail when the summary digest is missing or stale.
|
| 201 |
+
|
| 202 |
+
The recipe runner writes the root-level files a reviewer should open first:
|
| 203 |
+
`recipe-run.json`, `audit.json`, and `claim-safety.md`. For page-level recipes with
|
| 204 |
+
`runner.output_audit`, execution runs that audit once per materialized page output
|
| 205 |
+
directory before scoring and records `output-audit/output-audit-run.json` plus
|
| 206 |
+
`output-audit/command-template.txt` and Markdown with commands, logs, return codes,
|
| 207 |
+
and available audit artifact hashes. The runner honors
|
| 208 |
+
`runner.output_audit.result_filename` when recording the per-page audit JSON; if it is
|
| 209 |
+
omitted, the default is `limbu-output-audit.json`. The root audit requires each
|
| 210 |
+
recorded per-page audit JSON to still exist with the recorded SHA-256. It also
|
| 211 |
+
requires the output-audit command template sidecar when `recipe-run.json` records an
|
| 212 |
+
output-audit command, rejects a missing or stale sidecar, and replays each per-page
|
| 213 |
+
output-audit command from that template using the recorded output directory. It
|
| 214 |
+
also replays the recorded stdout/stderr log SHA-256 values for each per-page audit
|
| 215 |
+
command. It requires `output-audit/output-audit-run.md` and checks that the Markdown
|
| 216 |
+
report still reflects the recorded count, per-page output audit rows, log hashes, and
|
| 217 |
+
recorded audit artifact hashes.
|
| 218 |
+
The report summarizes what was scored, the GT trust
|
| 219 |
+
metadata, frozen/unseen declarations, leakage status, micro-CER metrics, output-audit
|
| 220 |
+
status, warnings, and review artifact locations. The root report is audited against
|
| 221 |
+
`recipe-run.json`, the score eval-run JSON, and `metrics.json`, with mode-specific
|
| 222 |
+
expectations for recognizer oracle-crop and page-level OCR score facts, so stale
|
| 223 |
+
top-level summaries fail audit.
|
| 224 |
+
The recipe-root audit also checks that `materialize/output-command-run.json` matches
|
| 225 |
+
the materialization summary embedded in `recipe-run.json`, that
|
| 226 |
+
`materialize/command-template.txt` and `materialize/output-command-run.md` match the
|
| 227 |
+
recorded command/output summary, that materialization timestamp, machine platform/
|
| 228 |
+
Python metadata, invocation, per-record command, and stdout/stderr log paths and
|
| 229 |
+
SHA-256 values are present and replay against current log bytes. It also replays
|
| 230 |
+
materialization record commands from the recorded command template using the recorded
|
| 231 |
+
recognizer or page materialization fields; if those fields are missing, the root audit
|
| 232 |
+
fails instead of skipping command replay. For recognizer roots, the root `predictions.txt` path and hash must be the
|
| 233 |
+
same prediction artifact recorded by the score eval-run JSON. For page roots,
|
| 234 |
+
each materialization record must carry the source page image path/hash, the recorded
|
| 235 |
+
hash is rechecked against current bytes, `outputs.documents_dir` must match the score
|
| 236 |
+
eval-run `documents_dir`, and materialized `document.json` outputs must live under that directory. Recipe-root materialization
|
| 237 |
+
model/checkpoint/config/dictionary metadata is also checked against the score eval-run
|
| 238 |
+
metadata.
|
| 239 |
+
If materialization fails before a score directory exists, keep
|
| 240 |
+
`materialize/output-command-run-error.json`: it carries the timestamp, machine,
|
| 241 |
+
invocation, model/checkpoint/config/dictionary metadata, command template, and label
|
| 242 |
+
or eval-manifest path/hash used for the failed attempt. If the recipe wrapper fails,
|
| 243 |
+
keep `recipe-run-error.json`: it records the timestamp, machine, command, recipe
|
| 244 |
+
registry hash, selected recipe validation, root, output directory, exception type,
|
| 245 |
+
message, and traceback.
|
| 246 |
+
The underlying recognizer or page score directory also writes its own
|
| 247 |
+
`claim-safety.md`, so direct scorer runs are reviewable without a recipe wrapper.
|
| 248 |
+
|
| 249 |
+
Audit the finished run directory before reporting the number:
|
| 250 |
+
|
| 251 |
+
```bash
|
| 252 |
+
uv run python tools/audit_eval_run.py \
|
| 253 |
+
--run-dir outputs/evals/<run-id> \
|
| 254 |
+
--out outputs/evals/<run-id>/audit.json \
|
| 255 |
+
--require-audit-pass
|
| 256 |
+
```
|
| 257 |
+
|
| 258 |
+
The audit recomputes recognizer row edits and aggregate micro-CER from
|
| 259 |
+
`score/predictions.tsv`; a metric file that is internally ratio-consistent but does
|
| 260 |
+
not match the emitted GT/prediction rows must fail audit. The run machine metadata
|
| 261 |
+
must include concrete `platform` and `python` values, and `normalization_profile`
|
| 262 |
+
must match the scorer-supported `NFC_preserve_whitespace` profile used by the audit
|
| 263 |
+
recomputation path. The audit also recomputes `normalization_summary` from
|
| 264 |
+
`predictions.tsv` for recognizer runs and `text-pairs.tsv` for page runs; those
|
| 265 |
+
fields count raw GT/reference and prediction strings changed by NFC, so
|
| 266 |
+
normalization-heavy scores remain visible instead of being hidden behind normalized
|
| 267 |
+
CER. The audit
|
| 268 |
+
also reconstructs `predictions.tsv` row identity, GT text, prediction text,
|
| 269 |
+
prediction score, crop hash, language/script/domain/GT/split, and row
|
| 270 |
+
claim-eligibility fields from the recorded label, prediction, and manifest sources.
|
| 271 |
+
The same file carries
|
| 272 |
+
language/script/domain/GT/split, claim-eligibility, and crop image SHA-256 values, so
|
| 273 |
+
recognizer slice metrics, claim-eligibility provenance, and crop-image identity are
|
| 274 |
+
recomputed from the emitted row artifact, not trusted as separate aggregates.
|
| 275 |
+
`worst-lines.csv` is also audited against those canonical prediction rows, including
|
| 276 |
+
crop hashes, trust metadata, claim eligibility, and source-document/page/bbox
|
| 277 |
+
provenance, so a failure sample cannot be made more claim-friendly by editing the
|
| 278 |
+
review artifact after scoring.
|
| 279 |
+
Recognizer label-line count, scored-line count, matched-line count, unmatched label
|
| 280 |
+
count, unmatched prediction count, and prediction match rate are recomputed from the
|
| 281 |
+
recorded source label and prediction files.
|
| 282 |
+
Recognizer `normalizer_counts` in `metrics.json` and `eval-run.json` are also checked
|
| 283 |
+
against the recorded summary/datasheet `counts`, so defaulted pack metadata cannot be
|
| 284 |
+
hidden by editing both score JSON files after scoring.
|
| 285 |
+
Recognizer and page summary/datasheet files are semantically audited too: if a
|
| 286 |
+
hash-valid summary declares language, script, domain, GT status, split, eval-pack
|
| 287 |
+
version, frozen/unseen status, default claim eligibility, or task metadata that
|
| 288 |
+
contradicts the scored run metadata, the audit fails.
|
| 289 |
+
Run-level language/script/domain/GT/split lists in `eval-run.json` are also checked
|
| 290 |
+
against the unique values in `predictions.tsv`.
|
| 291 |
+
`predictions.tsv` also carries source-document/page/bbox provenance columns, and the
|
| 292 |
+
audit reconstructs those values from the recorded label/manifest sources.
|
| 293 |
+
For page runs, the audit also recomputes `line-alignment.csv` row contents from the
|
| 294 |
+
scored `document.json` and reference JSON, including predicted/reference line IDs,
|
| 295 |
+
line bboxes, text, IoU, and per-line text CER, then recomputes line detection counts,
|
| 296 |
+
precision/recall/F1, mean IoU, and aggregate metric means from `line-alignment.csv`
|
| 297 |
+
and `page-metrics.csv`.
|
| 298 |
+
The audit also reconstructs `page-metrics.csv` row identity from the eval manifest,
|
| 299 |
+
resolved `document.json`, embedded `document.source_path`, scorer metadata defaults,
|
| 300 |
+
and parsed reference fields, so
|
| 301 |
+
language/script/domain/GT/split, source-document/page/bbox provenance,
|
| 302 |
+
`claim_eligible`, and `reference_fields` cannot be changed after scoring without
|
| 303 |
+
failing audit.
|
| 304 |
+
Run-level language/script/domain/GT/split lists in `page-eval-run.json` must match
|
| 305 |
+
the unique values in `page-metrics.csv`.
|
| 306 |
+
`reference-artifacts.tsv` carries the exact reference path, SHA-256, scoreable
|
| 307 |
+
reference fields, text hash, and reference trust metadata for each scored page, and
|
| 308 |
+
the audit recomputes it from the eval manifest and reference files.
|
| 309 |
+
`worst-pages.csv` is audited against `page-metrics.csv` plus
|
| 310 |
+
`reference-artifacts.tsv`, including input/document hashes, language/script/domain/GT
|
| 311 |
+
metadata, claim eligibility, source-document/page/bbox provenance, reference hashes,
|
| 312 |
+
and reference trust fields.
|
| 313 |
+
`page-metrics.csv` carries raw page text edit/ref counts and exact text-match flags,
|
| 314 |
+
`text-pairs.tsv` carries the exact prediction/reference page text used for those
|
| 315 |
+
counts, and the audit recomputes the text-pair artifact from scored `document.json`
|
| 316 |
+
text plus reference text before rolling it into page text micro-CER,
|
| 317 |
+
text exact-accuracy totals, and per-language/script/domain/GT/split page slices.
|
| 318 |
+
It also records source page image and `document.json` SHA-256 values, which the audit
|
| 319 |
+
rechecks so a page score cannot point at mutated OCR outputs after the fact.
|
| 320 |
+
Train/dev/eval leakage reports are recomputed from the recorded train manifests and
|
| 321 |
+
canonical eval rows/manifests, including checked state, row counts, and overlap sets.
|
| 322 |
+
For page-derived crop training or checkpoint-selection data, include `source_sample_id`
|
| 323 |
+
on train rows; scorer and audit treat overlap between train `source_sample_id` and eval
|
| 324 |
+
page `sample_id` as hard leakage even when crop image hashes and line text hashes do not
|
| 325 |
+
match the full page row directly.
|
| 326 |
+
Native spot-check calibration summaries embedded in run and metric JSON are checked
|
| 327 |
+
against the recorded calibration artifact.
|
| 328 |
+
Metadata contradictions are recomputed from manifest/reference provenance signals and
|
| 329 |
+
canonical row domain/GT fields.
|
| 330 |
+
Recognizer provenance coverage and page row metadata coverage counts are recomputed
|
| 331 |
+
from their source manifests, and page reference coverage is recomputed from
|
| 332 |
+
`page-metrics.csv`. Page manifest/scored/missing/claim-eligible counts are recomputed
|
| 333 |
+
from the eval manifest, document outputs, references, and claim-eligibility metadata.
|
| 334 |
+
Recognizer and page score directories also carry deterministic bootstrap uncertainty
|
| 335 |
+
summaries in `metrics.json`; the audit recomputes those intervals from
|
| 336 |
+
`predictions.tsv` or `page-metrics.csv`. Claim-safe runs currently require at least
|
| 337 |
+
30 scored rows/pages, so one-page smoke checks and tiny oracle-crop packs must remain
|
| 338 |
+
non-claim-safe even when their point CER is perfect.
|
| 339 |
+
Recognizer and page score directories also carry script inventory summaries in
|
| 340 |
+
`metrics.json` and run JSON; the audit recomputes observed codepoints/graphemes from
|
| 341 |
+
recognizer GT rows or page reference text and verifies the inventory file hash. A
|
| 342 |
+
missing inventory file, a stale inventory summary, or missing required
|
| 343 |
+
codepoints/graphemes remains a claim-safety blocker even when CER is low.
|
| 344 |
+
They also carry `dictionary_coverage` in `metrics.json`; the audit rehashes the
|
| 345 |
+
recorded dictionary file, recomputes the non-whitespace GT/reference codepoints, and
|
| 346 |
+
blocks claim-safe status when the ground truth contains codepoints outside the
|
| 347 |
+
dictionary vocabulary. `metrics.md` and `claim-safety.md` must surface the same
|
| 348 |
+
dictionary coverage, normalization-change, and zero-length GT/reference counters,
|
| 349 |
+
and the audit treats stale Markdown as an output-directory integrity failure.
|
| 350 |
+
Recognizer metrics record `zero_length_gt_lines`, and page metrics record
|
| 351 |
+
`zero_length_text_references`; the audit recomputes both from canonical rows and
|
| 352 |
+
blocks claim-safe status when any scored GT/reference text is empty.
|
| 353 |
+
The audit also derives claim-safety blockers from durable run facts, so a tampered
|
| 354 |
+
`claim_safe=true` field without matching warnings still fails if the artifacts show
|
| 355 |
+
metadata contradictions, a missing manifest or summary/datasheet, an unrun leakage
|
| 356 |
+
check, leakage overlaps, low recognizer match rate, unsafe GT/domain status, missing
|
| 357 |
+
provenance, missing uncertainty evidence, missing script inventory evidence, missing
|
| 358 |
+
required script inventory items, zero-length GT/reference text, too-small claim-facing
|
| 359 |
+
sample size, recognizer row-level claim ineligibility, or incomplete page coverage.
|
| 360 |
+
The direct and recipe-root `claim-safety.md` reports are also checked against their
|
| 361 |
+
JSON run and metric facts, using recognizer or page score fields according to
|
| 362 |
+
`eval_mode`, including score timestamp, machine, command provenance, hashed
|
| 363 |
+
checkpoint/config/dictionary artifact identity, normalization-change,
|
| 364 |
+
dictionary-coverage, script-inventory gaps, metadata contradictions, uncertainty
|
| 365 |
+
intervals, zero-length GT/reference counters, and per-language/script/domain/GT/split
|
| 366 |
+
slice metrics, so stale or aggregate-only reviewer-facing summaries fail audit instead
|
| 367 |
+
of merely having the right section headings. Direct and recipe-root score reports must
|
| 368 |
+
also list the review artifacts a reviewer needs to open, including `command.txt`,
|
| 369 |
+
canonical row artifacts, worst-example CSVs, sample-review CSV/HTML files, confusions,
|
| 370 |
+
and page alignment/reference artifacts.
|
| 371 |
+
For recipe-root audits, `recipe.json` must still match the selected recipe in the
|
| 372 |
+
hashed source registry and `recipe-validation.json` must still match
|
| 373 |
+
`recipe-run.json`; the embedded validation is also recomputed from the current recipe
|
| 374 |
+
registry and eval-pack audit. Recipe-run timestamp, machine, command, and root output
|
| 375 |
+
pointers are audited too. Root `audit.json` and the embedded recipe-run audit summary must
|
| 376 |
+
preserve current score-audit issues, warnings, claim-safe status, and score-dir
|
| 377 |
+
identity, and the recipe-run score summary must point to and hash the current
|
| 378 |
+
`score/metrics.json` plus the mode-specific score run file (`score/eval-run.json`
|
| 379 |
+
for recognizer roots, `score/page-eval-run.json` for page roots). The root train-manifest
|
| 380 |
+
path/hash list must also match the score eval-run leakage inputs, and the hashed
|
| 381 |
+
recipe registry must be under the declared recipe root. Do not hand-edit those files
|
| 382 |
+
after scoring.
|
| 383 |
+
For page-level Limbu book-candidate recipes that use
|
| 384 |
+
`tools/score_limbu_book_ocr_candidate.py`, the recipe validator also audits the
|
| 385 |
+
declared `evidence` chain. It reads the score wrapper, admission report, metrics, and
|
| 386 |
+
review-coverage report; verifies recorded SHA-256 values; requires champion metrics;
|
| 387 |
+
requires review-coverage counts for missed and extra lines; and blocks candidates with
|
| 388 |
+
champion regressions or uncovered review regions. It also parses the recipe
|
| 389 |
+
`command_template` and requires the scorer command to bind the same `--book-output`,
|
| 390 |
+
`--book-audit`, `--champion-metrics`, and `--review-coverage-report` paths recorded in
|
| 391 |
+
the evidence chain. A command string or free-text claim blocker is not enough evidence
|
| 392 |
+
for current-best registration, and stale evidence cannot be attached to a weaker replay
|
| 393 |
+
command.
|
| 394 |
+
The score-level `metrics.md` report is checked against `metrics.json` for core
|
| 395 |
+
counts, CER values, slices, reference coverage, metric means, and warnings.
|
| 396 |
+
Recognizer `unmatched.tsv` row identity, label image hashes, prediction scores, and
|
| 397 |
+
source-level match/count metrics are recomputed from
|
| 398 |
+
the recorded source label and prediction files, recognizer `confusions.json` is
|
| 399 |
+
recomputed from canonical predictions, page `confusions.json` is recomputed from
|
| 400 |
+
canonical `text-pairs.tsv` prediction/reference text, and page `missing.tsv` is
|
| 401 |
+
recomputed from the eval manifest, document outputs, and reference files.
|
| 402 |
+
Worst-example CSVs, reviewer CSV order and fields, and reviewer HTML samples are also
|
| 403 |
+
checked against the canonical scored rows, so stale or hand-edited failure samples do
|
| 404 |
+
not satisfy reviewability.
|
| 405 |
+
Completed score artifacts are checked for duplicate scored sample IDs, image paths,
|
| 406 |
+
document paths, and missing-row keys; duplicate IDs are not only a pre-normalization
|
| 407 |
+
pack failure.
|
| 408 |
+
|
| 409 |
+
For paper-claim numbers, also add `--require-claim-safe`.
|
| 410 |
+
|
| 411 |
+
## Recognizer Oracle-Crop Eval
|
| 412 |
+
|
| 413 |
+
Use this manual path for line-crop recognizers only. Do not use it for page-level
|
| 414 |
+
OCR claims.
|
| 415 |
+
|
| 416 |
+
### Manual Path
|
| 417 |
+
|
| 418 |
+
1. Normalize the eval pack into the canonical schema.
|
| 419 |
+
|
| 420 |
+
```bash
|
| 421 |
+
uv run python tools/normalize_recognizer_eval_pack.py \
|
| 422 |
+
--label-file data/<pack>/eval.txt \
|
| 423 |
+
--data-dir data/<pack> \
|
| 424 |
+
--manifest-file data/<pack>/manifest.jsonl \
|
| 425 |
+
--summary-file data/<pack>/summary.json \
|
| 426 |
+
--out outputs/eval-packs/<language>-<pack-version> \
|
| 427 |
+
--dataset <dataset-id> \
|
| 428 |
+
--language <language> \
|
| 429 |
+
--script <script> \
|
| 430 |
+
--domain real_print \
|
| 431 |
+
--gt-status machine_converted \
|
| 432 |
+
--split eval \
|
| 433 |
+
--eval-pack-version <pack-version> \
|
| 434 |
+
--frozen-eval-pack \
|
| 435 |
+
--unseen-for-selection \
|
| 436 |
+
--no-claim-eligible
|
| 437 |
+
```
|
| 438 |
+
|
| 439 |
+
If the normalizer reports missing source document, page, bbox, GT status, or claim
|
| 440 |
+
eligibility, do not hide it with defaults. Fix the pack or keep the result
|
| 441 |
+
non-claim-safe.
|
| 442 |
+
If a recognizer manifest row declares an image `sha256`, the scorer verifies that it
|
| 443 |
+
matches the current crop bytes. A stale declared hash fails scoring and fails later
|
| 444 |
+
audit if an older score directory points at a mutated manifest.
|
| 445 |
+
|
| 446 |
+
2. Build train/dev leakage manifests for every model family used to select or
|
| 447 |
+
train the checkpoint.
|
| 448 |
+
|
| 449 |
+
```bash
|
| 450 |
+
uv run python tools/build_recognizer_leakage_manifest.py \
|
| 451 |
+
--label data/<train-pack>/train.txt::data/<train-pack>::train \
|
| 452 |
+
--label data/<train-pack>/eval.txt::data/<train-pack>::dev \
|
| 453 |
+
--out outputs/leakage/<model-family>-train-dev.jsonl \
|
| 454 |
+
--summary outputs/leakage/<model-family>-train-dev.summary.json \
|
| 455 |
+
--source <model-family> \
|
| 456 |
+
--language <language> \
|
| 457 |
+
--script <script> \
|
| 458 |
+
--domain real_print \
|
| 459 |
+
--gt-status machine_converted
|
| 460 |
+
```
|
| 461 |
+
|
| 462 |
+
Use `--text-only` only when the corpus intentionally has no images. A text-only
|
| 463 |
+
manifest can catch text leakage, but it cannot catch duplicate crop images.
|
| 464 |
+
If crops were cut from a page eval candidate, the leakage manifest must preserve the
|
| 465 |
+
origin page as `source_sample_id`; otherwise a line-crop pack can look clean under exact
|
| 466 |
+
image/text hash checks while still being selected from the eval page.
|
| 467 |
+
|
| 468 |
+
3. Materialize predictions through a recorded command.
|
| 469 |
+
|
| 470 |
+
For repo candidate model cards, omit `--command` and provide `--model-config`:
|
| 471 |
+
|
| 472 |
+
```bash
|
| 473 |
+
uv run python tools/run_eval_output_command.py \
|
| 474 |
+
--mode recognizer \
|
| 475 |
+
--model-config outputs/<model-package>/model-card.json \
|
| 476 |
+
--out outputs/evals/<run-id>/materialize \
|
| 477 |
+
--system-name <system-name> \
|
| 478 |
+
--model-name <model-name> \
|
| 479 |
+
--checkpoint <local-checkpoint-file> \
|
| 480 |
+
--config <local-training-or-inference-config-file> \
|
| 481 |
+
--dictionary <local-dictionary-file> \
|
| 482 |
+
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
|
| 483 |
+
--data-dir outputs/eval-packs/<language>-<pack-version> \
|
| 484 |
+
--prediction-file outputs/evals/<run-id>/predictions.txt \
|
| 485 |
+
--timeout-seconds 3600
|
| 486 |
+
```
|
| 487 |
+
|
| 488 |
+
For external PaddleOCR/HF runners, pass `--command` explicitly. The command must
|
| 489 |
+
write `{prediction_file}` in `image<TAB>prediction<TAB>confidence` format and may
|
| 490 |
+
use only `{label_file}`, `{data_dir}`, `{prediction_file}`, and `{out_dir}`.
|
| 491 |
+
|
| 492 |
+
```bash
|
| 493 |
+
uv run python tools/run_eval_output_command.py \
|
| 494 |
+
--mode recognizer \
|
| 495 |
+
--command 'bash scripts/predict_external_recognizer.sh --labels {label_file} --data {data_dir} --out {prediction_file}' \
|
| 496 |
+
--out outputs/evals/<run-id>/materialize \
|
| 497 |
+
--system-name <system-name> \
|
| 498 |
+
--model-name <model-name> \
|
| 499 |
+
--checkpoint <local-checkpoint-file> \
|
| 500 |
+
--config <local-training-or-inference-config-file> \
|
| 501 |
+
--dictionary <local-dictionary-file> \
|
| 502 |
+
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
|
| 503 |
+
--data-dir outputs/eval-packs/<language>-<pack-version> \
|
| 504 |
+
--prediction-file outputs/evals/<run-id>/predictions.txt \
|
| 505 |
+
--timeout-seconds 3600
|
| 506 |
+
```
|
| 507 |
+
|
| 508 |
+
4. Score the predictions.
|
| 509 |
+
|
| 510 |
+
```bash
|
| 511 |
+
uv run python tools/run_recognizer_eval.py \
|
| 512 |
+
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
|
| 513 |
+
--data-dir outputs/eval-packs/<language>-<pack-version> \
|
| 514 |
+
--prediction-file outputs/evals/<run-id>/predictions.txt \
|
| 515 |
+
--out outputs/evals/<run-id>/score \
|
| 516 |
+
--manifest-file outputs/eval-packs/<language>-<pack-version>/manifests/recognizer-eval.jsonl \
|
| 517 |
+
--summary-file outputs/eval-packs/<language>-<pack-version>/summary.json \
|
| 518 |
+
--script-inventory-file outputs/eval-packs/<language>-<pack-version>/script-inventory.json \
|
| 519 |
+
--train-manifest outputs/leakage/<model-family>-train-dev.jsonl \
|
| 520 |
+
--model-name <model-name> \
|
| 521 |
+
--checkpoint <local-checkpoint-file> \
|
| 522 |
+
--config <local-training-or-inference-config-file> \
|
| 523 |
+
--dictionary <local-dictionary-file> \
|
| 524 |
+
--language <language> \
|
| 525 |
+
--script <script> \
|
| 526 |
+
--domain real_print \
|
| 527 |
+
--gt-status machine_converted \
|
| 528 |
+
--split eval \
|
| 529 |
+
--eval-pack-version <pack-version> \
|
| 530 |
+
--frozen-eval-pack \
|
| 531 |
+
--unseen-for-selection \
|
| 532 |
+
--no-selected-checkpoint-on-test \
|
| 533 |
+
--no-claim-eligible \
|
| 534 |
+
--min-match-rate 0.99
|
| 535 |
+
```
|
| 536 |
+
|
| 537 |
+
Only add `--require-claim-safe` when every warning is expected to be absent. If
|
| 538 |
+
the GT status is `machine_converted_spot_checked`, also pass
|
| 539 |
+
`--spotcheck-calibration-file`. Recipe runs can declare the same artifact under
|
| 540 |
+
`eval_pack.spotcheck_calibration_file` so the calibration evidence is part of the
|
| 541 |
+
versioned recipe instead of an invisible operator-only flag.
|
| 542 |
+
For a claim-safe recognizer run, `--checkpoint`, `--config`, and `--dictionary`
|
| 543 |
+
must point to readable local files. The scorer records their SHA-256 hashes in
|
| 544 |
+
`eval-run.json`, and `tools/audit_eval_run.py` re-verifies those hashes and checks
|
| 545 |
+
that the visible run fields still match the hashed artifact identities.
|
| 546 |
+
The scorer also records whether all non-whitespace GT codepoints are present in the
|
| 547 |
+
dictionary; missing GT codepoints are a claim-safety warning and are independently
|
| 548 |
+
recomputed during audit.
|
| 549 |
+
The same audit checks hash-valid summary/datasheet metadata against the direct run's
|
| 550 |
+
recorded language/script/domain/GT/split, version, frozen/unseen, default
|
| 551 |
+
claim-eligibility, and task metadata.
|
| 552 |
+
`--script-inventory-file` must also point to a readable local JSON file; it is hashed in
|
| 553 |
+
`eval-run.json`, summarized in `metrics.json`, and audited against the emitted GT rows.
|
| 554 |
+
|
| 555 |
+
After scoring, audit the score directory:
|
| 556 |
+
|
| 557 |
+
```bash
|
| 558 |
+
uv run python tools/audit_eval_run.py \
|
| 559 |
+
--run-dir outputs/evals/<run-id>/score \
|
| 560 |
+
--out outputs/evals/<run-id>/score/audit.json \
|
| 561 |
+
--require-audit-pass
|
| 562 |
+
```
|
| 563 |
+
|
| 564 |
+
### Same-Pack Recognizer Comparisons
|
| 565 |
+
|
| 566 |
+
When a paper claim compares two recognizers, do not compare separate aggregate numbers
|
| 567 |
+
by eye. Score both systems on the same eval pack, audit both score directories, then
|
| 568 |
+
run the paired comparison tool:
|
| 569 |
+
|
| 570 |
+
```bash
|
| 571 |
+
uv run python tools/compare_recognizer_eval_runs.py \
|
| 572 |
+
--candidate-score-dir outputs/evals/<candidate-run>/score \
|
| 573 |
+
--baseline-score-dir outputs/evals/<baseline-run>/score \
|
| 574 |
+
--candidate-name <candidate-name> \
|
| 575 |
+
--baseline-name <baseline-name> \
|
| 576 |
+
--out outputs/evals/<comparison-run> \
|
| 577 |
+
--min-pairs 30 \
|
| 578 |
+
--require-claim-safe
|
| 579 |
+
```
|
| 580 |
+
|
| 581 |
+
The tool pairs rows by `sample_id`, checks that eval-pack identity including
|
| 582 |
+
`pack_content_sha256` and row GT identity match, and emits `comparison.json`,
|
| 583 |
+
`comparison.md`, `paired-rows.tsv`, and
|
| 584 |
+
`command.txt`. It reports codepoint and grapheme micro-CER deltas, paired
|
| 585 |
+
wins/losses/ties, deterministic paired-row bootstrap confidence intervals for the
|
| 586 |
+
improvement, and a two-sided exact sign-test p-value. A comparison is claim-safe only
|
| 587 |
+
when both input score directories are claim-safe, the rows are fully paired on the same
|
| 588 |
+
eval pack, the bootstrap lower bound is above zero, the sign test passes the configured
|
| 589 |
+
alpha, the candidate actually improves the baseline, and comparison timestamp/machine/
|
| 590 |
+
command provenance is present and internally consistent.
|
| 591 |
+
|
| 592 |
+
Audit the comparison directory before using it in a claim:
|
| 593 |
+
|
| 594 |
+
```bash
|
| 595 |
+
uv run python tools/compare_recognizer_eval_runs.py \
|
| 596 |
+
--audit-only \
|
| 597 |
+
--comparison-dir outputs/evals/<comparison-run> \
|
| 598 |
+
--require-claim-safe
|
| 599 |
+
```
|
| 600 |
+
|
| 601 |
+
For page-level OCR system comparisons, use the page comparison tool instead of the
|
| 602 |
+
recognizer comparison tool:
|
| 603 |
+
|
| 604 |
+
```bash
|
| 605 |
+
uv run python tools/compare_page_ocr_eval_runs.py \
|
| 606 |
+
--candidate-score-dir outputs/evals/<candidate-page-run> \
|
| 607 |
+
--baseline-score-dir outputs/evals/<baseline-page-run> \
|
| 608 |
+
--candidate-name <candidate-name> \
|
| 609 |
+
--baseline-name <baseline-name> \
|
| 610 |
+
--out outputs/evals/<page-comparison-run> \
|
| 611 |
+
--min-pairs 30 \
|
| 612 |
+
--require-claim-safe
|
| 613 |
+
```
|
| 614 |
+
|
| 615 |
+
It pairs pages by `sample_id`, checks page eval-pack identity including
|
| 616 |
+
`pack_content_sha256` plus page/reference text identity, and emits `comparison.json`,
|
| 617 |
+
`comparison.md`, `paired-pages.tsv`, and `command.txt` for page text
|
| 618 |
+
codepoint/grapheme CER deltas, paired wins/losses/ties, paired-page bootstrap
|
| 619 |
+
intervals, and a two-sided exact sign-test p-value. Audit it the same way:
|
| 620 |
+
|
| 621 |
+
```bash
|
| 622 |
+
uv run python tools/compare_page_ocr_eval_runs.py \
|
| 623 |
+
--audit-only \
|
| 624 |
+
--comparison-dir outputs/evals/<page-comparison-run> \
|
| 625 |
+
--require-claim-safe
|
| 626 |
+
```
|
| 627 |
+
|
| 628 |
+
## Native-Reader Spot Check
|
| 629 |
+
|
| 630 |
+
Machine-converted labels may be scored, but they are not human-verified evidence.
|
| 631 |
+
Use this path to create auditable calibration evidence.
|
| 632 |
+
|
| 633 |
+
```bash
|
| 634 |
+
uv run python tools/prepare_native_spotcheck_bundle.py \
|
| 635 |
+
--label-file outputs/eval-packs/<language>-<pack-version>/eval.txt \
|
| 636 |
+
--data-dir outputs/eval-packs/<language>-<pack-version> \
|
| 637 |
+
--manifest-file outputs/eval-packs/<language>-<pack-version>/manifests/recognizer-eval.jsonl \
|
| 638 |
+
--prediction-file outputs/evals/<run-id>/predictions.txt \
|
| 639 |
+
--out outputs/spotchecks/<language>-<pack-version> \
|
| 640 |
+
--sample-size 50 \
|
| 641 |
+
--seed 20260619 \
|
| 642 |
+
--title '<language> native-reader spot check'
|
| 643 |
+
```
|
| 644 |
+
|
| 645 |
+
After a reviewer exports the completed CSV:
|
| 646 |
+
|
| 647 |
+
```bash
|
| 648 |
+
uv run python tools/apply_native_spotcheck_reviews.py \
|
| 649 |
+
--review-file outputs/spotchecks/<language>-<pack-version>/reviewed-spotcheck.csv \
|
| 650 |
+
--out outputs/spotchecks/<language>-<pack-version>/calibration \
|
| 651 |
+
--language <language> \
|
| 652 |
+
--script <script> \
|
| 653 |
+
--domain real_print \
|
| 654 |
+
--source-gt-status machine_converted \
|
| 655 |
+
--min-usable-rows 30 \
|
| 656 |
+
--max-machine-grapheme-cer 0.02 \
|
| 657 |
+
--max-unusable-rate 0.05
|
| 658 |
+
```
|
| 659 |
+
|
| 660 |
+
If the calibration output says `claim_evidence_eligible=false`, do not mark the
|
| 661 |
+
pack `machine_converted_spot_checked`. Completed rows without reviewer identity
|
| 662 |
+
are not eligible calibration evidence; fix the review export rather than
|
| 663 |
+
overriding the gate. The scorer also requires the calibration artifact's language,
|
| 664 |
+
script, domain, `source_gt_status=machine_converted`, and
|
| 665 |
+
`recommended_pack_gt_status=machine_converted_spot_checked` to match the scored run.
|
| 666 |
+
|
| 667 |
+
## Page-Level OCR Eval
|
| 668 |
+
|
| 669 |
+
Use this path for end-to-end page claims. Recognizer-only scores cannot support a
|
| 670 |
+
page OCR claim because detection, crop lineage, and reading order are absent.
|
| 671 |
+
|
| 672 |
+
1. Materialize page outputs through a recorded command.
|
| 673 |
+
|
| 674 |
+
```bash
|
| 675 |
+
uv run python tools/run_eval_output_command.py \
|
| 676 |
+
--mode page \
|
| 677 |
+
--model-config outputs/<page-model-package>/model-card.json \
|
| 678 |
+
--out outputs/page-evals/<run-id>/materialize \
|
| 679 |
+
--system-name <system-name> \
|
| 680 |
+
--model-name <model-name> \
|
| 681 |
+
--checkpoint <local-checkpoint-file> \
|
| 682 |
+
--config <local-training-or-inference-config-file> \
|
| 683 |
+
--dictionary <local-dictionary-file> \
|
| 684 |
+
--eval-manifest data/<page-eval-pack>/manifests/eval.jsonl \
|
| 685 |
+
--documents-dir outputs/page-evals/<run-id>/documents \
|
| 686 |
+
--timeout-seconds 3600
|
| 687 |
+
```
|
| 688 |
+
|
| 689 |
+
For an external runner, pass `--command` and write `{document_json}` for each
|
| 690 |
+
input page. Page commands may use only `{input_path}`, `{sample_id}`, `{output_dir}`,
|
| 691 |
+
`{document_json}`, `{documents_dir}`, and `{out_dir}`.
|
| 692 |
+
|
| 693 |
+
2. Score saved `document.json` outputs.
|
| 694 |
+
|
| 695 |
+
```bash
|
| 696 |
+
uv run python tools/run_page_ocr_eval.py \
|
| 697 |
+
--eval-manifest data/<page-eval-pack>/manifests/eval.jsonl \
|
| 698 |
+
--documents-dir outputs/page-evals/<run-id>/documents \
|
| 699 |
+
--ocr-output-kind live_detector \
|
| 700 |
+
--out outputs/page-evals/<run-id>/score \
|
| 701 |
+
--summary-file data/<page-eval-pack>/summary.json \
|
| 702 |
+
--script-inventory-file data/<page-eval-pack>/script-inventory.json \
|
| 703 |
+
--model-name <model-name> \
|
| 704 |
+
--checkpoint <local-checkpoint-file> \
|
| 705 |
+
--config <local-training-or-inference-config-file> \
|
| 706 |
+
--dictionary <local-dictionary-file> \
|
| 707 |
+
--eval-pack-version <pack-version> \
|
| 708 |
+
--frozen-eval-pack \
|
| 709 |
+
--unseen-for-selection \
|
| 710 |
+
--no-selected-checkpoint-on-test \
|
| 711 |
+
--train-manifest outputs/leakage/<model-family>-train-dev.jsonl
|
| 712 |
+
```
|
| 713 |
+
|
| 714 |
+
For direct live-detector page scoring, claim-safe mode requires a passing output audit
|
| 715 |
+
for every scored page output directory. By default the scorer looks for
|
| 716 |
+
`limbu-output-audit.json` with `passed=true` beside each `document.json`; use
|
| 717 |
+
`--require-output-audit`, `--output-audit-filename`, and
|
| 718 |
+
`--output-audit-passed-field` when scoring a different page-output audit format.
|
| 719 |
+
The page auditor replays those per-page audit artifacts from `page-metrics.csv` and
|
| 720 |
+
fails if a recorded `page_output_audit` summary no longer matches the current files,
|
| 721 |
+
so deleting or editing an output audit after scoring invalidates the score directory.
|
| 722 |
+
|
| 723 |
+
A paper-safe page claim requires reference text plus page/line layout references
|
| 724 |
+
that let the scorer report detection/alignment/reading-order behavior. If line
|
| 725 |
+
bboxes are absent, the result is not a full page-level OCR claim. Each page
|
| 726 |
+
manifest row must also carry explicit `language`, `script`, `domain`, and
|
| 727 |
+
`gt_status` metadata plus source-document, page, bbox, and claim-eligibility
|
| 728 |
+
provenance; CLI defaults are recorded as fallbacks, not as row-level claim evidence.
|
| 729 |
+
The page summary must carry a `pack_content_sha256` over the canonical page row,
|
| 730 |
+
input-image, reference-artifact, reference-text, and provenance identity; claim-safe
|
| 731 |
+
scoring rejects summaries whose digest is missing or stale.
|
| 732 |
+
Page manifest `sample_id` and `image_path` values must be unique,
|
| 733 |
+
and any declared row `sha256` must match the actual input image bytes; duplicate IDs,
|
| 734 |
+
duplicate input images, or stale image hashes fail before scoring. If page GT status
|
| 735 |
+
is `machine_converted_spot_checked`, pass
|
| 736 |
+
`--spotcheck-calibration-file`; without an eligible calibration artifact the
|
| 737 |
+
page result is not claim-safe.
|
| 738 |
+
For a claim-safe page run, `--checkpoint`, `--config`, and `--dictionary` must point
|
| 739 |
+
to readable local files whose SHA-256 hashes can be recorded and audited.
|
| 740 |
+
`--script-inventory-file` must point to a readable local JSON file; the scorer records
|
| 741 |
+
which required codepoints/graphemes were covered by reference text and the auditor
|
| 742 |
+
recomputes that summary from the eval manifest and references.
|
| 743 |
+
`--ocr-output-kind` must identify what kind of document outputs were scored. Use
|
| 744 |
+
`live_detector` for normal end-to-end page OCR outputs, `oracle_line_boxes` for
|
| 745 |
+
diagnostic runs where reference/PDF line boxes are fed to recognizers, and
|
| 746 |
+
`sidecar_fixture` only for tests or synthetic fixtures. For Limbu, the detector
|
| 747 |
+
diagnostic requires scoring the same frozen page pack once as `live_detector` and once
|
| 748 |
+
as `oracle_line_boxes`, then comparing the two score directories with
|
| 749 |
+
`tools/compare_page_ocr_eval_runs.py`.
|
| 750 |
+
|
| 751 |
+
Audit the page score directory:
|
| 752 |
+
|
| 753 |
+
```bash
|
| 754 |
+
uv run python tools/audit_eval_run.py \
|
| 755 |
+
--run-dir outputs/page-evals/<run-id>/score \
|
| 756 |
+
--out outputs/page-evals/<run-id>/score/audit.json \
|
| 757 |
+
--require-audit-pass
|
| 758 |
+
```
|
| 759 |
+
|
| 760 |
+
## Claim-Safe Gates
|
| 761 |
+
|
| 762 |
+
A recognizer or page result is not paper-safe unless all of these are true:
|
| 763 |
+
|
| 764 |
+
- Eval pack is frozen, versioned, and declared unseen for model/checkpoint
|
| 765 |
+
selection.
|
| 766 |
+
- The score records a manifest and summary/datasheet for the eval pack.
|
| 767 |
+
- The score records a script inventory file, its hash, observed codepoint/grapheme
|
| 768 |
+
coverage, and no missing required codepoints or graphemes.
|
| 769 |
+
- Manifest row IDs and input image paths are unique, recognizer manifest image
|
| 770 |
+
paths/basenames cannot collide with basename-based prediction matching, and declared
|
| 771 |
+
source-image hashes match the current files; stale recognizer crop hashes, duplicate
|
| 772 |
+
page manifest IDs or image paths, stale page manifest image hashes, and duplicate
|
| 773 |
+
completed score artifact IDs fail audit.
|
| 774 |
+
- Every scored row has source-document, page, bbox, language, script, domain,
|
| 775 |
+
split, GT status, and claim eligibility metadata, and every scored row is claim
|
| 776 |
+
eligible.
|
| 777 |
+
- GT status is `human_verified` or backed by eligible native-reader calibration
|
| 778 |
+
as `machine_converted_spot_checked`.
|
| 779 |
+
- Rows declared `human_verified` carry human review evidence such as `reviewed_by`,
|
| 780 |
+
`review_artifact`, double-review, agreement, or correction-rate metadata; the trust
|
| 781 |
+
label alone is not claim evidence.
|
| 782 |
+
- The domain is real target material, not synthetic or transliterated evidence.
|
| 783 |
+
- Row/reference provenance does not contradict the declared domain or GT status:
|
| 784 |
+
synthetic/transliterated provenance cannot be claimed as real material, and
|
| 785 |
+
machine-converted provenance cannot be claimed as `human_verified`.
|
| 786 |
+
- Train/dev leakage manifests were passed and checked.
|
| 787 |
+
- Model artifacts are hash-verifiable: recognizer and page runs require checkpoint,
|
| 788 |
+
config, and dictionary files, and
|
| 789 |
+
the run fields must match the hashed artifact identity entries.
|
| 790 |
+
- Model identity is recorded: recognizer and page runs require a non-empty model name,
|
| 791 |
+
and page runs require a non-empty system name.
|
| 792 |
+
- No hard leakage exists by sample ID, image hash, image path, or combined
|
| 793 |
+
sample hash.
|
| 794 |
+
- The prediction match rate is at least the configured threshold.
|
| 795 |
+
- No scored recognizer GT row or page text reference is zero-length.
|
| 796 |
+
- Headline reporting is per language/script/domain/GT status, not a pooled
|
| 797 |
+
average hiding weak languages, and every emitted language/script/domain/GT/split
|
| 798 |
+
slice has at least the claim-facing minimum scoring units.
|
| 799 |
+
- Same-pack recognizer comparisons use `tools/compare_recognizer_eval_runs.py` and pass
|
| 800 |
+
its paired bootstrap/sign-test gates; separate aggregate score directories are not
|
| 801 |
+
enough evidence for a comparative claim.
|
| 802 |
+
- The score directory contains `claim-safety.md` with `What Was Scored`, `Ground
|
| 803 |
+
Truth And Pack`, `Metrics`, and `Warnings` sections.
|