# Research State: Nepal Language Pack OCR Updated: 2026-08-09 ## Current Goal The end goal is an OCR system covering the multiple languages of the Nepal pack, built out of Paddle, **with Limbu and Newar (Nepal Bhasa) as the first two focus languages** (user steer, 2026-07-04). The system should accept real book/page images or PDFs for any language in the pack and output editable Unicode text with page, line, crop, confidence, and provenance evidence preserved. Limbu stays the proven template — it has the strongest recognizer and the deepest harvest. Newar is the deliberate second: it has its own native script (Prachalit), real cultural demand, and is currently the weakest trained recognizer, so it exercises every part of the method that Limbu already proved. Every stage that was built Limbu-specific must generalize: harvest, silver-label extraction, recognizer training, script routing, page/book OCR, and eval. The product is the full pipeline: page detection, script-aware routing to per-script recognizers, target-region and furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and review artifacts for hard lines. YOLO line detection stays a diagnostic/probe; Paddle-first is the production story. ## The Language Pack Two inventories exist and they are not the same thing: - The registry `languages/nepal-v1.json` tracks 18 target languages (17 Nepal + English) with scripts, fonts, dictionaries, and corpus status. - The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found **50 normalized languages (53 raw labels) in 400 articles over ~7 months** (see the harvest section). Labels include all registry languages plus Chamling Rai, Bahing, Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo, Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi, Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more. Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा, ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) — `tools/build_gorkhapatra_language_pack_inventory.py` owns that normalization and the label→registry mapping. Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded 2026-06-18 scorecard in `docs/RECOGNIZER_GOAL_PROGRESS.md`): - **Limbu/Sirijonga 0.041 real-print** (champion: `runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23`, cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval). - **8 Devanagari languages 0.075–0.129 real-print** (Majhi, Magar, Sunuwar-Deva, Magahi, Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer. - **Sunuwar (Jenticha) 0.121 real-crop** after the same-font retrain (0.342 synth-only). - **Tirhuta 0.343 and Newa 0.613 synthetic-only** — weak, transliteration-fed, honest fails; Newa needs real printed-register Nepal Lipi, not more pixels. - **Kirat Rai (70 real crops) and Lepcha (132 real crops)** blocked on legacy-font converter coverage outside their eval editions; **Gurung Khema** has no real print anywhere (revival script) — synthetic-only by nature, claim-scoped accordingly. The near-term pack claim should therefore be script-family shaped: one Sirijonga recognizer, one shared Devanagari recognizer covering the many Devanagari-written languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa, Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval crops exist. **Focus order (2026-07-04 steer): Limbu first, Newar second.** Newar state: the Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own synthetic eval; transliteration-fed training failed; real printed-register Nepal Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20 Gorkhapatra pages; `newar` added to the registry (Devanagari primary + Prachalit secondary; audit passes); a frozen by-date pack built at `outputs/newar-devanagari-silver-pack-v1/` (train 4,096 / dev 1,528 / test 1,139 lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently verified; pack sha256 `1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85`); and the **first real-print Devanagari-Newar baseline** was scored locally (CPU export route, `tools/score_recognizer_pack.py`, per `docs/LOCAL_RECOGNIZER_SCORING.md`): the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at **codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158** (n=1,139 silver lines — machine-converted labels, diagnostic, not claim-grade). The same-day warm-start fine-tune (**newar-ft-v1**: deva-v2 base, LR 2e-5 cosine, 60 epochs, batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance `ashish-experiments`, ~68 min; artifacts + train log at `outputs/aws-newar-deva-ft-v1/`) scores the same frozen test at **codepoint CER 0.0465, grapheme CER 0.0955, exact-line 0.245** — a 25/27% relative CER drop and +55% exact-line over the deva-v2 baseline, scored through the identical local CPU path. **Newar's grapheme CER now sits inside deva-v2's own-language range (0.075–0.129): the second focus language reached Devanagari-pack parity from zero in one day.** Same claim boundary: silver machine-converted labels, diagnostic, not claim-grade until native review. Next Newar lever: native Prachalit (real printed-register data), and page-level Newar eval for the region-filtering track. ## Source Rule For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata only. Use `https://gorkhapatraonline.com/publications/naya-nepal` to find the language, date, and article, but do not use the blog/card image for OCR, training, or evaluation when the full paper exists. The authoritative source is the matching Gorkhapatra epaper PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus the text layer extracted from that same PDF page, with PDF URL/hash, page number, rendered image hash, extraction command/version, font evidence, and converter/map hash recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with `src/ocrtech/limbu_legacy.py` + `data/mappings/limbu-legacy/Limbu.map`. The same pattern — classify the text layer, then build an explicit converter only when the font requires it — replicates to every other language. Manual transcription is the fallback for no usable text layer, not the default. ## Getting More Gorkhapatra Pages (the specified harvest recipe) All numbers below were verified live on 2026-07-04. **Archive size.** The Naya Nepal publication archive paginates at `https://gorkhapatraonline.com/publications/naya-nepal?page=N` with 5 articles per page and runs to page ~352 before drying up: **~1,750 language-page articles total** (about 2 per day going back years). The 60-page crawl already yielded 300 items across 53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly older dates. **Epaper endpoint behavior (the critical constraint).** The date-range endpoint `https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra` **caps results at 30 PDFs per query and ignores `page=N` pagination**. A wide window silently drops most dates — the 2026-07-04 six-window discovery pass aligned only 111/300 items for exactly this reason, with unmatched dates interleaved among matched ones. **Seven-day windows return complete daily coverage (7/7 verified).** Always harvest with weekly windows. **The recipe** (one command; add one `--epaper-url` per week to cover the publication date span being harvested): ```bash env PYTHONPATH=src uv run --python 3.11 --with pymupdf \ ocrtech audit-gorkhapatra-language-pages \ --out outputs/gorkhapatra-language-pages-full- \ --publication-url https://gorkhapatraonline.com/publications/naya-nepal \ --publication-pages 80 \ --epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \ --epaper-url '...one per week, going back...' \ --download-pdfs \ --render-pages \ --timeout-seconds 180 ``` The command aligns every publication card (all languages at once — alignment is language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL), searches pages for the language-page markers (`भाषा पृष्ठ` and its legacy-encoding form `efiff k[i7`), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with **400 publication items, 195 aligned dates, 394 rendered language-page hits, zero warnings** at `outputs/gorkhapatra-language-pages-full-2026-07-04/`. Batch 2 (2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more items: **798 items, 294 aligned dates, 589 further page hits**, at `outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/`. The merged normalized inventory (`tools/build_gorkhapatra_language_pack_inventory.py`) shows **56 languages, 983 rendered language-page hits, most languages with ~20–27 pages spanning 14 months**, at `outputs/language-pack-inventory-merged-2026-07-04/`. For comparison, the entire previous state of the project had 8 usable pages for one language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least March 2024 — verified the epaper endpoint serves complete weeks there) is another ~2× and stays month-batchable with the same recipe. **Scaling to the full archive.** Full depth is ~700+ unique dates ≈ 35 GB of PDFs. Do not do it in one run: batch by month (4–5 weekly windows + the matching publication-page depth), run `extract-gorkhapatra-language-page-pdf-text` per batch, then keep rendered pages, extracted text, and hashes; PDFs can be pruned after extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume (`cdjk@100.110.148.43:/Volumes/TRANSCEND`) is the right home for a full-depth mirror. **Per-language downstream.** After harvest, run per language: ```bash env PYTHONPATH=src uv run --python 3.11 --with pymupdf \ ocrtech extract-gorkhapatra-language-page-pdf-text \ \ --candidate-language