prosperious's picture
Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified
|
Raw History Blame Contribute Delete
23.8 kB

Research State: Nepal Language Pack OCR

Updated: 2026-08-09

Current Goal

The end goal is an OCR system covering the multiple languages of the Nepal pack, built out of Paddle, with Limbu and Newar (Nepal Bhasa) as the first two focus languages (user steer, 2026-07-04). The system should accept real book/page images or PDFs for any language in the pack and output editable Unicode text with page, line, crop, confidence, and provenance evidence preserved. Limbu stays the proven template — it has the strongest recognizer and the deepest harvest. Newar is the deliberate second: it has its own native script (Prachalit), real cultural demand, and is currently the weakest trained recognizer, so it exercises every part of the method that Limbu already proved. Every stage that was built Limbu-specific must generalize: harvest, silver-label extraction, recognizer training, script routing, page/book OCR, and eval. The product is the full pipeline: page detection, script-aware routing to per-script recognizers, target-region and furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and review artifacts for hard lines. YOLO line detection stays a diagnostic/probe; Paddle-first is the production story.

The Language Pack

Two inventories exist and they are not the same thing:

  • The registry languages/nepal-v1.json tracks 18 target languages (17 Nepal + English) with scripts, fonts, dictionaries, and corpus status.
  • The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found 50 normalized languages (53 raw labels) in 400 articles over ~7 months (see the harvest section). Labels include all registry languages plus Chamling Rai, Bahing, Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo, Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi, Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more. Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा, ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) — tools/build_gorkhapatra_language_pack_inventory.py owns that normalization and the label→registry mapping.

Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded 2026-06-18 scorecard in docs/RECOGNIZER_GOAL_PROGRESS.md):

  • Limbu/Sirijonga 0.041 real-print (champion: runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23, cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval).
  • 8 Devanagari languages 0.075–0.129 real-print (Majhi, Magar, Sunuwar-Deva, Magahi, Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer.
  • Sunuwar (Jenticha) 0.121 real-crop after the same-font retrain (0.342 synth-only).
  • Tirhuta 0.343 and Newa 0.613 synthetic-only — weak, transliteration-fed, honest fails; Newa needs real printed-register Nepal Lipi, not more pixels.
  • Kirat Rai (70 real crops) and Lepcha (132 real crops) blocked on legacy-font converter coverage outside their eval editions; Gurung Khema has no real print anywhere (revival script) — synthetic-only by nature, claim-scoped accordingly.

The near-term pack claim should therefore be script-family shaped: one Sirijonga recognizer, one shared Devanagari recognizer covering the many Devanagari-written languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa, Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval crops exist.

Focus order (2026-07-04 steer): Limbu first, Newar second. Newar state: the Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own synthetic eval; transliteration-fed training failed; real printed-register Nepal Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20 Gorkhapatra pages; newar added to the registry (Devanagari primary + Prachalit secondary; audit passes); a frozen by-date pack built at outputs/newar-devanagari-silver-pack-v1/ (train 4,096 / dev 1,528 / test 1,139 lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently verified; pack sha256 1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85); and the first real-print Devanagari-Newar baseline was scored locally (CPU export route, tools/score_recognizer_pack.py, per docs/LOCAL_RECOGNIZER_SCORING.md): the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158 (n=1,139 silver lines — machine-converted labels, diagnostic, not claim-grade). The same-day warm-start fine-tune (newar-ft-v1: deva-v2 base, LR 2e-5 cosine, 60 epochs, batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance ashish-experiments, ~68 min; artifacts + train log at outputs/aws-newar-deva-ft-v1/) scores the same frozen test at codepoint CER 0.0465, grapheme CER 0.0955, exact-line 0.245 — a 25/27% relative CER drop and +55% exact-line over the deva-v2 baseline, scored through the identical local CPU path. Newar's grapheme CER now sits inside deva-v2's own-language range (0.075–0.129): the second focus language reached Devanagari-pack parity from zero in one day. Same claim boundary: silver machine-converted labels, diagnostic, not claim-grade until native review. Next Newar lever: native Prachalit (real printed-register data), and page-level Newar eval for the region-filtering track.

Source Rule

For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata only. Use https://gorkhapatraonline.com/publications/naya-nepal to find the language, date, and article, but do not use the blog/card image for OCR, training, or evaluation when the full paper exists. The authoritative source is the matching Gorkhapatra epaper PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus the text layer extracted from that same PDF page, with PDF URL/hash, page number, rendered image hash, extraction command/version, font evidence, and converter/map hash recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with src/ocrtech/limbu_legacy.py + data/mappings/limbu-legacy/Limbu.map. The same pattern — classify the text layer, then build an explicit converter only when the font requires it — replicates to every other language. Manual transcription is the fallback for no usable text layer, not the default.

Getting More Gorkhapatra Pages (the specified harvest recipe)

All numbers below were verified live on 2026-07-04.

Archive size. The Naya Nepal publication archive paginates at https://gorkhapatraonline.com/publications/naya-nepal?page=N with 5 articles per page and runs to page 352 before drying up: **1,750 language-page articles total** (about 2 per day going back years). The 60-page crawl already yielded 300 items across 53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly older dates.

Epaper endpoint behavior (the critical constraint). The date-range endpoint https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra caps results at 30 PDFs per query and ignores page=N pagination. A wide window silently drops most dates — the 2026-07-04 six-window discovery pass aligned only 111/300 items for exactly this reason, with unmatched dates interleaved among matched ones. Seven-day windows return complete daily coverage (7/7 verified). Always harvest with weekly windows.

The recipe (one command; add one --epaper-url per week to cover the publication date span being harvested):

env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
  ocrtech audit-gorkhapatra-language-pages \
    --out outputs/gorkhapatra-language-pages-full-<date> \
    --publication-url https://gorkhapatraonline.com/publications/naya-nepal \
    --publication-pages 80 \
    --epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \
    --epaper-url '...one per week, going back...' \
    --download-pdfs \
    --render-pages \
    --timeout-seconds 180

The command aligns every publication card (all languages at once — alignment is language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL), searches pages for the language-page markers (भाषा पृष्ठ and its legacy-encoding form efiff k[i7), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with 400 publication items, 195 aligned dates, 394 rendered language-page hits, zero warnings at outputs/gorkhapatra-language-pages-full-2026-07-04/. Batch 2 (2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more items: 798 items, 294 aligned dates, 589 further page hits, at outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/. The merged normalized inventory (tools/build_gorkhapatra_language_pack_inventory.py) shows 56 languages, 983 rendered language-page hits, most languages with ~20–27 pages spanning 14 months, at outputs/language-pack-inventory-merged-2026-07-04/. For comparison, the entire previous state of the project had 8 usable pages for one language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least March 2024 — verified the epaper endpoint serves complete weeks there) is another ~2× and stays month-batchable with the same recipe.

Scaling to the full archive. Full depth is ~700+ unique dates ≈ 35 GB of PDFs. Do not do it in one run: batch by month (4–5 weekly windows + the matching publication-page depth), run extract-gorkhapatra-language-page-pdf-text per batch, then keep rendered pages, extracted text, and hashes; PDFs can be pruned after extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume (cdjk@100.110.148.43:/Volumes/TRANSCEND) is the right home for a full-depth mirror.

Per-language downstream. After harvest, run per language:

env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
  ocrtech extract-gorkhapatra-language-page-pdf-text \
    <pack-manifest.jsonl> \
    --candidate-language <label> \
    --target-font <font> \
    --out outputs/<language>-pdf-text-<date>

then classify each page's text layer (native Unicode / legacy font / Devanagari transcription / Latin / none) and only then decide converter vs manual transcription.

Font/text-layer facts already established (30-page, 15-language extraction at outputs/gorkhapatra-multilingual-language-pages-current-2026-06-24-pdf-text/):

  • The body font of non-Limbu language pages is nayanepal — a legacy 8-bit Preeti-family encoding, not Unicode (8,845 of 11,791 spans; the text layer reads k[i7 ;+of]hs where the page shows पृष्ठ संयोजक). The converter already exists: nepal_ttf2utf.convert_devanagari(font="nayanepal") in the sibling repo /Users/cdjk/github/llm/nepal-ttf2utf (Preeti base map + nayanepal extension glyphs + NFC, ~97–99% clean on real Gorkhapatra pages), already used by tools/build_gorkhapatra_devanagari_real.py for the 79k-row real Devanagari corpus. The remaining leverage is wiring it per language: run it over each harvested language page and emit per-language silver line packs with the same faithfulness gate (clean-Unicode allow-set, conversion_gap accounting, leakage checks) that the Devanagari pack builder already implements. The same sibling repo also hosts the limbu, kiratrai, lepcha, and sunuwar legacy converters.
  • Tibetan-script fonts (TibetanMachine, MonlamUniOuChan5, Qomolangma-*) appear on Mugal and related pages, but every such span has empty extracted text (verified 2026-07-04 across all 2026-06-24 samples) — Tibetan-script pages are visual-only, like the known Limbu 2082-11-12 date. They need OCR bootstrap or manual transcription; no converter is possible.
  • Gorkhapatra-*, Nagarik-*, TimesNewRoman*, MyriadPro-* are masthead/furniture fonts — useful as non-target filters for target-region selection.
  • NamdhinggoSILL appears only on Limbu dates (the existing Limbu path).

What We Built (Limbu template, still true)

The Gorkhapatra alignment path: publication-page discovery, epaper-date matching, PDF download/rendering, page-marker search, PDF-native span/font extraction, review sheets, reference templates, and dashboard visualization. The expanded Limbu harvest has 9 discovered dates, 8 usable PDF-text-layer pages, 2,441 line rows, 2,430 usable silver labels (outputs/limbu-gorkhapatra-pdf-backed-silver-2026-07-03-v1/).

The Limbu recognizer training/eval path: 48,507-row combined pack; real-heavy fine-tune champion at codepoint CER 0.011336 / grapheme 0.024736 / exact-line 0.729 on the frozen 255-line silver eval. Synthetic hardening lessons: targeted+Unicode packs regressed; v3 dictionary warm-start catastrophically failed (~0.98 CER); keep the real-heavy v2 champion; future gains come from real/book-domain harvesting or a deliberate rehead, not blind synth rows.

The page/book OCR loop: local + HF Space (spaces/limbu-ocr/, deployed private at https://huggingface.co/spaces/ampixa/limbu-ocr) with line OCR, Paddle Page OCR, Paddle Book OCR, image-line fallback, runtime checks, annotated pages, and crops.

Page-level diagnostics on the 8-page silver pack: Rows LTR + y/conf70 filter scores page cp CER 0.414330, line F1 0.963009, matched-line cp CER 0.026039, order accuracy 0.993176. Recognition on matched lines is strong; full-page CER is dominated by page reconstruction, target-region selection, and furniture filtering — that is the pipeline problem to solve, and it is language-independent.

Supervised line-detector infrastructure from CDC PDF geometry (287 pages, 26,160 annotations, mAP50 ≈ 0.921 on PDF-rendered val) — evidence the detector path works, not a solved product.

A claim-grade eval harness (versioned packs, hashes, provenance, normalization profiles, cp/grapheme CER, CIs, paired comparisons, leakage manifests) — parked as a guardrail; do not let it eat work blocks that should go to the pipeline.

Current Working State

The strongest component is the Sirijonga line recognizer. The weakest component is end-to-end page/book reconstruction: target-region selection, furniture filtering, line preservation, and ordering. Of the three Limbu-locked code spots, two were generalized on 2026-07-04 (Codex worker, tests green): _script_ratio in src/ocrtech/engines.py is now driven by the shared SCRIPT_CODEPOINT_RANGES table covering all pack scripts (devanagari, limbu/sirijonga, latin, sunuwar/jenticha, lepcha, kirat_rai, newa/prachalit, tirhuta, gurung_khema, ol_chiki, tibetan, with aliases), and the image-line filter in src/ocrtech/image_lines.py accepts any script key from that same table. The third spot fell the same day: the src/ocrtech/limbu_runtime.py preflight now takes required_script (default limbu, validated against the shared table; CLI --required-script), so a router card for any pack script can be preflighted. The engine classes themselves (CandidateEngine, PaddleOcrEngine, ScriptSelectCompositeEngine, build_image_line_document, structure.py reading order) are already parameterized — the rest of the generalization is registry-driven wiring, not a rewrite.

Apple MPS Recognizer Training Evidence (2026-08-09)

Stock PaddlePaddle is not the immediate Apple-GPU route. A direct PaddlePaddle 3.3.1 probe on k2 exposed no custom devices and rejected mps, metal, and custom_device:metal. Paddle's official PaddleCustomDevice MPS backend exists, but is currently a version-0.0.1 prototype with too small an operator surface to establish PP-OCRv5 training.

The practical path is an Apache-2.0 vendored subset of frotms/PaddleOCR2Pytorch pinned at c05307fbd61e575d55f0fef0334026c7536aec76: PPLCNetV3 scale 0.95 plus the SVTR/CTC branch of the PP-OCRv5 mobile recognizer. Dynamic pooling preserves the recognition width (T = input_width / 4), and a localized contiguity fix before the SVTR encoder's conv3 makes the model safe for the tested MPS runtimes. The model and log-softmax run on MPS; CTCLoss runs on CPU and autograd returns finite gradients to MPS for the optimizer update. The NRTR auxiliary branch is frozen and not part of this CTC loss, so this is not a complete reproduction of PaddleOCR's multi-loss training recipe.

One-step evidence is green on both Macs. The end-to-end mbp2 run used Python 3.11.15 and Torch 2.13.0: [4,3,48,160] -> [4,40,7], loss 29.883989, 582 finite MPS gradient tensors, and a non-zero optimizer update across synthetic Latin, Devanagari, and Limbu samples. Direct width-320 forward/backward/update checks also passed on mbp2 and on k2 with Torch 2.12. These cold one-step timings are diagnostics, not benchmarks.

The checkpoint boundary is strict: this backend writes PyTorch checkpoints, not Paddle .pdparams; conversion requires explicit tensor mapping and frozen inference-parity tests. mbp2 and k2 should run independent experiments with distinct seeds and run directories, not multi-host MPS DDP. The remaining proof is a multi-epoch real-manifest run with resume/reload, memory observation, and canonical held-out CER. See docs/PADDLEOCR_MPS.md for exact smoke artifacts, hashes, compatibility limits, and host guidance.

The Paddle Pipeline (whole pipeline, not just recognizers)

Target architecture, all stages Paddle-first and language-pack-parameterized by the per-language pipeline config configs/nepal-language-pack-pipeline-v1.json (50 languages: label, registry id, scripts, recognizer artifact, dictionary, converter, text-layer class, harvest counts — generated 2026-07-04 from the inventory):

  1. Capture/prep — existing prepare-limbu-capture path, already language-agnostic in behavior; rename/rescope.
  2. Detection — Paddle text detection as production default; YOLO line detector and morphology image-line boxes stay as diagnostics/rescue passes.
  3. Script identification & routing — generalize _script_ratio to a Unicode-block-driven scorer over the pack's script list (Devanagari, Limbu, Jenticha/Sunuwar, Lepcha, Kirat Rai, Newa, Tirhuta, Gurung Khema, Ol Chiki, Tibetan, Latin), then route each line to the script's recognizer from the registry.
  4. Recognition — per-script PP-OCRv5 recognizers (existing runs) with per-script dictionaries and input-width profiles (Sirijonga width-480 lesson).
  5. Target-region & furniture filtering — non-oracle region selection using layout plus font/language signals; furniture fonts and masthead geometry as negative evidence. This is the main open research item; score against the Rows LTR + y/conf70 baseline.
  6. Reading order & assembly — auto_layout selection (wide-column RTL vs rows) per page, conservative assembly stays diagnostic.
  7. Export — Unicode text, JSON with provenance, Markdown; review queues + dashboards per page (existing Limbu artifact shapes, renamed pack-wide).
  8. Eval guardrail — frozen real-page packs per language: page CER/WER, line P/R/F1, reading order, blank/extra/missed lines, review burden.

PP-StructureV3 and PaddleOCR-VL remain the documented fallback/teacher direction for hard layouts, per docs/NEPAL_MULTILINGUAL_OCR_GOAL.md.

Next Work

  1. Finish the 2026-07-04 full harvest and inventory Done 2026-07-04: 394 language-page hits, 50 languages, inventory manifests at outputs/language-pack-inventory-full-2026-07-04/.
  2. Per-language silver packs via the existing nayanepal converter Done 2026-07-04 (tools/build_language_pack_silver_line_packs.py, tests green): the batch-1 run produced 116,287 kept silver lines across 50 languages, zero errors, with per-line provenance (PDF hash, page, bbox, raw legacy bytes, converted text, image hash) at outputs/language-pack-silver-lines-2026-07-04/; spot-checked conversions read as clean, language-plausible Devanagari. The batch-2 run over the older 589 page hits kept 183,324 more lines (49 languages, zero errors) at outputs/language-pack-silver-lines-batch2-2026-07-04/ — 299,611 per-language-attributed silver lines total across the two batches (~2.4 GB of crops). Run the tool with uv run --python 3.11 --with pymupdf --with npttf2utf — nepal_ttf2utf needs the npttf2utf base-map package at runtime, and without it every span fails (cleanly, with per-span error records, but kept=0). Honest caveat: lines are page-attributed, not language-verified — a language page also carries Nepali furniture/boxes, and pages whose body is not nayanepal Devanagari (Urdu, the Tibetan-script pages) yield mostly furniture lines. A language-ID/lexicon filter before per-language recognizer training is the follow-up gate.
  3. Newar track (second focus language): add the Newar registry entry (Devanagari + Prachalit rows); build the Devanagari-written Newar recognizer eval/train split from the 6,800 harvested silver lines (by-date split, frozen test); keep native Prachalit as the separate longer arc — real printed-register Nepal Lipi harvesting, not more transliteration, is the lever.
  4. Classify Tibetan-script pages (Hyolmo/Mugal/Bhote) and decide converter vs native Unicode.
  5. Generalize the pipeline wiring Done 2026-07-04: all three Limbu-locked spots generalized (shared script table, filter keys, --required-script preflight), tests green; configs/nepal-language-pack-pipeline-v1.json holds the per-language wiring.
  6. Target-region/furniture filtering stage for Paddle page/book OCR, scored against the current baseline on the 8-page Limbu pack and at least one Devanagari-language pack (Newar is the natural candidate given the focus order).
  7. Only after page reconstruction improves, export high-quality real/silver line crops back into recognizer training, per language.
  8. Month-batch the archive back-harvest toward the full ~1,750-article depth, storing the mirror on the remote 2 TB volume.

Mistakes To Avoid

Do not use Naya Nepal blog images as training/eval evidence when the PDF exists. Do not use wide epaper date windows — the endpoint silently caps at 30 PDFs per query; weekly windows only. Do not treat unnormalized Gorkhapatra labels as distinct languages (थारु vs थारू). Do not treat the nayanepal text layer as Unicode — it is legacy Preeti-family and needs the converter before any silver label is real. Do not report Paddle norm_edit_dis as CER; real CER comes from tools/eval_metrics.py micro CER over canonical predictions. Do not treat synthetic data as claim evidence, and label Gurung Khema results as synthetic-by-nature. Do not promote the targeted+Unicode checkpoint or v3 dictionary warm-starts. Do not spend long blocks on generic eval-harness work unless it directly improves the book/page OCR loop. Do not frame video-to-page work as OCR scope; reduce video to stills first. Do not present page OCR as solved: matched-line recognition is strong, but full-page CER is still dominated by reconstruction and region selection.