# Research State: Nepal Language Pack OCR Updated: 2026-09-24 ## Devanagari recognizer fine-tune on k2 (2026-09-24) Goal (owner steer): make the Devanagari route of the live demo (`ampixa.thapa-ashish.com.np`, k2 OCR worker) read citizenship-certificate-style and general documents properly. Resource inventory: `docs/DEVANAGARI_RESOURCES_INVENTORY.md`. - **Base**: official `devanagari_PP-OCRv5_mobile_rec` (568-entry dictionary incl. all Latin letters and digits); replaced our `deva-v2` on the demo the same morning. - **Training path**: Paddle CPU training on k2 (Apple M2 Ultra) runs at ~2 img/s, so the same network is trained with the `transformers` PyTorch port on MPS (~70-90 img/s), exported to ONNX, and served through PaddleX `create_model(engine="onnxruntime")`. Code: `tools/deva_rec_torch/`. Preprocessing is bit-matched to PaddleX (98.8% line parity). - **Data**: real Gorkhapatra print re-cut into PDF-aligned chunks (`tools/build_gorkhapatra_devanagari_chunks.py`, edition-disjoint: test 2082-03-24, 2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11); synthetic citizenship/office/real-text lines in 44 OFL font families (`tools/render_devanagari_form_synth.py`); NVIDIA OCR-Synthetic-Multilingual-v1 English (CC BY 4.0) plus Liberation/DejaVu/Lato English renders. - **Converter bug found and fixed**: npttf2utf 0.3.7 mis-decodes the nayanepal alternate क्र ligature `j|m` (-> व्र, and प -> फ / stray `m`). 3,426 of 114,452 spans (3%) were affected (कार्यक्रम, प्रक्रिया, शुक्रबार, ...). Fixed in the chunk builder (`RAW_FIXES`); labels of all earlier Gorkhapatra-derived packs (incl. `data/gorkhapatra-devanagari-real`, the Newar and language-pack silver packs built from the same converter) carry the bug. **Upstream fix still needed in nepal-ttf2utf.** - **Results, corrected labels, served path (PaddleX), codepoint CER / exact line** (`~/ocr-deva-ft/preds/final_comparison_v3.json` on k2): | Set | official | v2 | **v3 (deployed 13:07)** | |---|---|---|---| | E1v4 real chunks, held-out editions (n=3000) | 5.13% / 0.47 | 0.69% / 0.91 | 0.57% / 0.93 | | E7 real chunks containing क्र (n=530) | 5.52% / 0.33 | 4.28% / 0.18 | 0.86% / 0.89 | | E1b real full lines, independent labels (n=1500) | 7.69% / 0.13 | 2.62% / 0.53 | 2.57% / 0.54 | | E2 Newar test (n=1139) | 9.11% / 0.06 | 3.53% / 0.35 | 3.54% / 0.35 | | E3 synthetic citizenship-ne, unseen fonts (n=800) | 6.78% / 0.56 | 0.84% / 0.93 | 0.75% / 0.94 | | E3 synthetic office-ne, unseen fonts (n=300) | 7.77% / 0.54 | 0.62% / 0.94 | 0.92% / 0.92 | | E3 synthetic citizenship-en (n=300) | 1.94% / 0.83 | 0.53% / 0.96 | 0.64% / 0.95 | | E4 English, unseen Noto Sans (n=120) | 2.57% / 0.62 | 1.58% / 0.73 | 1.64% / 0.72 | | E4 mixed Nepali/English (n=120) | 6.78% / 0.36 | 2.88% / 0.68 | 2.67% / 0.70 | | E6 NVIDIA English, held-out pages (n=419) | 10.6% / 0.71 | 2.99% / 0.84 | 2.84% / 0.84 | | E5 handwriting probe (n=450) | 64.7% | 60.5% | 61.0% (unusable) | v2 learned the converter's क्र->व्र error (402 of 530 E7 predictions contained व्र); v3, a 6,198-step continuation on the corrected `chunks-v4`, removes it (2 of 530, same as the official model). Deployed as `~/ampixa-demo/models/ocr/space/models/devanagari-ft-v3`; v2 and the official model remain on disk for rollback. E3 is synthetic from the same generator as training (different seed, held-out fonts) and overstates real-certificate accuracy; no real citizenship certificate has been tested. Real labels are PDF-text silver, not human-reviewed. Page OCR at 144 dpi garbles small text; at 288 dpi it reads cleanly. - **Next**: real citizenship samples (with consent) for a true eval; handwriting data (none licensed yet); upstream the `j|m` fix into nepal-ttf2utf and rebuild the other Gorkhapatra-derived packs; the demo's "Rows LTR" reading order interleaves newspaper columns. ### Page pipeline on real government letters (2026-09-24, smoke test) Two public notices from ocr.gov.np (a scanned Lalitpur District Court letter tilted ~2°, and a Department of Industry letter), rendered at 200 dpi; 43 printed lines transcribed by hand (`tools/deva_rec_torch/page_probe/`: GT, probe scripts, results; PDFs on k2 `~/ocr-deva-ft/layout-probe/`; not a frozen pack, not claim-grade). Same detector config and v3 recognizer as the k2 worker; micro CER via `tools/eval_metrics.py`: | Crop of each detected line | cp CER | grapheme CER | exact | |---|---|---|---| | worker today: axis-aligned bbox of the quad + 12 px | 21.4% | 31.7% | 0.40 | | rotated min-area quad (PaddleX `CropByPolys`), no pad | 3.6% | 5.5% | 0.49 | | rotated quad + 6 px | **2.9%** | **4.2%** | **0.65** | | rotated quad + 12 px | 8.1% | 13.3% | 0.42 | - **The worker's crop, not the recognizer, is the dominant error on real letters.** `_box_from_polygon` flattens the detector quad to an upright box; on a 2° tilt a full-width line spans ~40 px vertically, so the crop swallows neighbouring lines (court-letter body lines read at confidence 0.24-0.57). Upstream PaddleOCR uses rotated crops; our port dropped them. Line-level evals (E1-E7) cannot see this. Not yet deployed: needs a page-level eval. - **PP-DocLayoutV3 on the same pages**: QR codes found (image, 0.95/0.96); emblems found (header_image); **0 of 2 stamps labelled `seal`** (court seal -> header_image 0.65; the Industry stamp printed over the title not detected at all); signatures merged into the printed-name `text` block (0.52); no class for handwritten fill-ins; several printed lines (subject, dates, reference numbers) not boxed at the default threshold. Use layout to mask non-text regions, never to decide what gets OCR'd. - Signature/stamp crops produce junk text at recognizer confidence 0.35-0.71; damaged but real lines (tick mark, strike-through) score 0.74-0.82, so confidence can flag but not drop. - `train.py` now supports `--guard-dev` (print/synthetic/English dev sets may not regress more than `--guard-tol` over step 0); the plain dev mean let a 5-point handwriting gain hide a 1-point print loss. `run_train_v4.sh` on k2 uses it. ## Paper-rewrite verification pass (2026-07-19): corrections to this document While rewriting `paper/DRAFT.md` to cover the language pack, every number was re-verified against on-disk artifacts. Four values recorded earlier in this file do NOT match any surviving local artifact and must not be cited: - **Newar "both-sides-normalized cp CER 0.0441"** — no local artifact; the omac rescore output was never mirrored. The surviving verified score is **0.04655** cp / 0.09569 grapheme / 0.2450 exact (artifact-bearing labels, `outputs/newar-ft-v1-verify-omac-2026-07-07/metrics.json`, n=1,139). - **Ol Chiki "public headline cp CER 0.0961 / n=358"** — no local score artifact exists for the 358-row public-issue subset. Locally verified: v2 on the 772-line v3 pack scores **0.08264** cp / 0.2034 exact (`outputs/olchiki-v2-run-2026-07-07/score-real-print-v3pack/metrics.json`); the documented public-issue historical diagnostic remains 0.1018 (n=366) per `docs/DATA_LICENSING.md`. The v3 358-row candidate was never scored. - **"299,611 per-language-attributed silver lines"** — refers to the lost pre-crash batches. The rebuilt batches on disk total **306,971** (122,882 + 184,089, `outputs/silver-lines-rebuild-b{1,2}-2026-07-05.log`). - **Ol Chiki v3 FT payload "5,330 entries"** — the payload's own assembly manifest says **5,307** total images (242 real train + 5,000 synth + 65 real dev). Also noted: the 8-language Devanagari scorecard (grapheme CER 0.075–0.129, 2026-06-18) lost its backing artifacts in the crash; it is text-only history now and the paper deliberately does not report it. ## Machine-death recovery (read this first, 2026-07-05) The Mac died 2026-07-04; the repo now lives at `/Volumes/TRANSCEND/recovery-backup-20260704-172213/github/llm/ocr-tech` (symlinked from the old `/Users/cdjk/github/llm/ocr-tech` path). **`outputs/` and `runs/` are lost**; code, `.git`, docs, `spaces/` model exports (sha256-verified, champion included), and most of `data/` survived. Every `outputs/…`/`runs/…` path below therefore refers to a lost artifact unless `docs/RECOVERY_LEDGER.md` maps it to a rebuilt one — read that ledger for the full survived/partial/lost inventory, recovery routes, and the rebuild policy (rebuilds go to new dated dirs). Batch-1 re-harvest launched 2026-07-05. Blocked on user credentials: AWS SSO (training checkpoints, newar-ft-v1 weights) and HF auth (Space cross-check). ## Overnight lane results (2026-07-16/17) These lanes delivered mechanically filtered silver substrate, not claim-grade training or evaluation data. The counts below establish file and archive parity; they do not replace native label review, edition-isolated leakage checks, or a real-domain score. The Sikkim pack at `outputs/sikkim-silver-packs-2026-07-16/` contains **9,743 Sunuwar/Jenticha clean real-page silver lines** and **3,699 Kirat Rai clean real-page silver lines**. In each case the `summary.json` clean-line count, on-disk `labels.txt` line count, and non-AppleDouble `.tgz` image-member count are identical. Sampled Sunuwar labels contain Jenticha characters in U+11BC0..U+11BFF, and sampled Kirat Rai labels contain Kirat Rai characters in U+16D40..U+16D7F, without non-ASCII characters outside the expected target block. Each summary records the eval manifests it checked. None of those eval editions occurs in the discovered source list, so the recorded `excluded_editions: []` is consistent rather than evidence of an unperformed check. Lepcha takes a different route. The diagnosis at `outputs/lepcha-zero-yield-diagnosis-2026-07-16/REPORT.md` shows that the new edition bodies are raster content and that the live PDF text is ordinary header/punctuation material, so there is no converter alias or byte-map fix that can recover the body. The converter lane is therefore closed and Lepcha moves to a synth-first recognizer route, retaining the **132-crop** real pack recorded in `data/lepcha-real-eval-pack/README.json` as a real evaluation anchor. That pack is still machine-labelled and human verification remains pending, so it is diagnostic rather than claim-grade. The Gorkhapatra full-depth backfill now covers every month from March 2024 through July 2025. The figures come from each month's `gorkhapatra-language-pages.json`; `silver-kept` comes from the corresponding silver summary and was independently matched to the `.tgz` member count. | Month | Items | Aligned | Hits | Silver-kept | | --- | ---: | ---: | ---: | ---: | | 2024-03 | 1,776 | 62 | 120 | 27,668 | | 2024-04 | 1,731 | 61 | 129 | 21,429 | | 2024-05 | 1,666 | 62 | 128 | 24,612 | | 2024-06 | 1,601 | 60 | 118 | 21,924 | | 2024-07 | 1,538 | 61 | 120 | 28,301 | | 2024-08 | 1,473 | 60 | 120 | 24,534 | | 2024-09 | 1,408 | 60 | 122 | 32,804 | | 2024-10 | 1,343 | 53 | 103 | 25,055 | | 2024-11 | 1,278 | 52 | 106 | 30,788 | | 2024-12 | 1,213 | 61 | 121 | 39,115 | | 2025-01 | 1,148 | 62 | 124 | 40,526 | | 2025-02 | 1,083 | 56 | 112 | 35,391 | | 2025-03 | 1,018 | 59 | 117 | 35,482 | | 2025-04 | 953 | 57 | 113 | 34,815 | | 2025-05 | 888 | 59 | 118 | 36,878 | | 2025-06 | 823 | 60 | 120 | 37,836 | | 2025-07 | 798 | 32 | 62 | 18,428 | | **Total** | **21,738** | **977** | **1,953** | **515,586** | All monthly audit warning arrays are empty, all archive counts match their silver summaries, and no source PDF remains under any monthly output. This is a delivered backfill substrate, not a statement that its page-attributed silver labels are language-verified. Detector remediation remains active. The scale-normalized tile build is staged/running on `omac`; the local evidence at `outputs/det-remediation-2026-07-16/REPORT.md` ends at the completed code, config, and handoff. The GPU gate remains closed until the returned build summary and tile manifest pass the recorded re-audit checklist. ## Current Goal The end goal is an OCR system covering the multiple languages of the Nepal pack, built out of Paddle, **with Limbu and Newar (Nepal Bhasa) as the first two focus languages** (user steer, 2026-07-04). The system should accept real book/page images or PDFs for any language in the pack and output editable Unicode text with page, line, crop, confidence, and provenance evidence preserved. Limbu stays the proven template — it has the strongest recognizer and the deepest harvest. Newar is the deliberate second: it has its own native script (Prachalit), real cultural demand, and is currently the weakest trained recognizer, so it exercises every part of the method that Limbu already proved. Every stage that was built Limbu-specific must generalize: harvest, silver-label extraction, recognizer training, script routing, page/book OCR, and eval. The product is the full pipeline: page detection, script-aware routing to per-script recognizers, target-region and furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and review artifacts for hard lines. YOLO line detection stays a diagnostic/probe; Paddle-first is the production story. ## The Language Pack Two inventories exist and they are not the same thing: - The registry `languages/nepal-v1.json` tracks 18 target languages (17 Nepal + English) with scripts, fonts, dictionaries, and corpus status. - The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found **50 normalized languages (53 raw labels) in 400 articles over ~7 months** (see the harvest section). Labels include all registry languages plus Chamling Rai, Bahing, Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo, Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi, Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more. Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा, ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) — `tools/build_gorkhapatra_language_pack_inventory.py` owns that normalization and the label→registry mapping. ### Newari / Nepal Bhasa writing-system boundary (2026-07-15) Newari support is explicitly four-profile, not a single generic "Newa" bucket: | Profile | Source encoding | OCR transcription | Current capability | | --- | --- | --- | --- | | Newar / Prachalit | Unicode Newa, U+11400..U+1147F | Native Newa | Route, training candidate, diagnostic checkpoint, and synthetic runtime smoke registered; no claim-grade real-domain CER | | Bhujimol / Bhujinmol | No separate Unicode script; configured as a Newa-codepoint style profile | Newa plus mandatory source-style metadata | Route, two acquired manuscript groups, and reviewer reference registered; zero qualified decisions, admitted labels, model, or CER | | Ranjana | Unencoded in Unicode 17.0 | Newa transcription plus mandatory source image and Ranjana metadata | Image route, two acquired manuscript groups, and reference font registered; zero qualified decisions, admitted labels, model, or CER | | Devanagari | Unicode Devanagari, U+0900..U+097F plus U+A8E0..U+A8FF | Native Devanagari | Route, diagnostic checkpoint/runtime smoke, and one permission-covered official Newa-edition acquisition candidate registered; source script and real-line quality remain unaudited | Unicode's current [Newa chapter](https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-13/) and [code chart](https://www.unicode.org/charts/PDF/U11400.pdf) define the encoded Newa route. WG2 document [N4555](https://www.unicode.org/wg2/docs/n4555.pdf) describes Bhujimol/Bhujinmol as graphically distinct and discusses possible plain-text unification with Newa; that does not justify visual-model reuse. Ranjana is still represented by a [preliminary encoding proposal](https://www.unicode.org/L2/L2023/23028-ranjana.pdf), so this project does not invent a Ranjana Unicode range. The canonical runtime profiles live in `src/ocrtech/newari_scripts.py`. The registry and book-OCR pipeline declare four separate recognizer references and preserve the source encoding/output policy in run metadata. The frozen audit at `outputs/newari-four-writing-system-support-audit-2026-07-15-r1/` passes all four structural bindings while reporting `model_ready=0/4` and `four_script_ocr_claim_ready=false`. "Supported" here means explicit input, profile, routing, and output semantics; each profile still needs at least 200 license-clean, pixel-label-audited, leakage-controlled real lines and a frozen real-domain evaluation before delivery or quality claims. The r2 license-first discovery audit records three Bhujimol and three Ranjana source candidates without admitting any pixels. The most promising Bhujimol item is [Asa Archives item 1147](https://archive.org/details/1147_20250817): its Internet Archive metadata has an uploader-applied CC0 URL, 63 scanned pages, and 325 automatically detected hOCR line boxes. Those boxes are acquisition-scale metadata, not labels. Uploader authority, institutional accession, exact-file SHA-256, qualified Bhujimol review, pixel faithfulness, and transcription mapping all remain open. For Ranjana, the Smithsonian's [Karandavyuha Sutra record](https://asia-archive.si.edu/object/S1991.161/) specifically marks image `FS-6314_06` CC0 and its IIIF metadata identifies Ranjana, but the [manifest](https://ids.si.edu/ids/manifest/FS-6314_06) contains only one canvas; the 77 physical folios are not 77 downloadable pages. British Library [EAP790/21/121](https://searcharchives.bl.uk/catalog/040-003461535) identifies 12 Bhujimol TIFFs but restricts access to research use, and [EAP1296/2/217](https://eap.bl.uk/archive-file/EAP1296-2-217) exposes 13 Ranjana images under noncommercial terms. The [Nepalbhasa Manuscripts in UK](https://www.nepalbhasamanuscript.org/) archive states that copies and transcriptions are openly accessible but supplies no explicit reuse grant. The frozen audit is `outputs/newari-source-candidate-audit-2026-07-15-r2/`; its pass means the catalogue is fail-closed, not that any source, model, or OCR claim is ready. The r3 provenance crosswalk matches item 1147 to the uploader's Asa Kuthi catalog spreadsheet at worksheet `aiStuidoOutput`, row 1170: DPN 1147, *Bhāsvati*, NRN `ज्योति - २७१`, RN 2919, Sanskrit/Nepalbhasa, N.S. 528, and 33.3 × 4.1 cm. The spreadsheet SHA-256 is `8484dbe95358b83d7d78d5d1752fafac5feeecfc007ccf5d1fe8b6ea3ce9bffd`. This disambiguates a separate row whose RN, rather than DPN, is 1147. It does not attest Bhujimol because the matched catalog script field is blank, and it does not prove that the Internet Archive uploader was authorized to apply CC0. The exact authorization and reuse questions are staged in `docs/NEWARI_ASA_1147_RIGHTS_REQUEST.md` but have not been sent. The r3 frozen audit therefore remains `PASS_CATALOG_INVENTORIED_ZERO_DATA_ADMITTED` with no review manifest ready. The r4 alternative-source sweep found a materially stronger Bhujimol route in Wellcome Collection. The official [manuscript record](https://wellcomecollection.org/works/jnsn9gjd) identifies `MS Indic epsilon 1`, and Wellcome's [editorial account of the object](https://wellcomecollection.org/stories/stories-of-asian-palm-leaf-manuscripts) explicitly says it uses Bhujimol. Its [complete IIIF manifest](https://iiif.wellcomecollection.org/presentation/v2/b14901560) has 217 unique canvases/image services, SHA-256 `596296750114948b39da2252b5a7b7c0738287dadcf7aed32939bfc29980c6d4`, and a Public Domain Mark statement that expressly permits commercial copying, modification, and distribution. Wellcome describes 208 leaves with only six illustrated. Five 2000-pixel-wide probes at canvases 1, 50, 100, 150, and 210 decoded successfully; canvas 1 was a cover/label view, while the other four showed dense multi-block palm-leaf text. This is not a script-expert or label audit. Because the current host is an 18GB Mac, the repository freezes only a metadata acquisition manifest and an omac handoff; it does not bulk-download the 217 canvases. The whole manuscript must remain on one side of any split, so a separate manuscript is still required for leakage-proof training. The r5 Ranjana follow-up resolved Wellcome `MS Indic delta 33` without inflating selected images into a complete manuscript. The official [physical record](https://wellcomecollection.org/works/bp979bf3) is closed-store only and has no digital item; the predictable [IIIF presentation URL](https://iiif.wellcomecollection.org/presentation/v2/b14901468) returns 404. Four exact digital-image records identify Ranjana and carry CC BY 4.0, but a prediction-free 1200-pixel probe shows one two-leaf overview, one partial-leaf environmental photograph, and two close-up miniature crops whose content overlaps the overview. The records therefore provide zero independent complete pages, not four. Their titles say AD 1653 while one description says transcribed in 1635; both values are retained. The frozen catalogues at `outputs/newari-wellcome-ranjana-source-catalog-2026-07-15-r5/` and `outputs/newari-source-candidate-audit-2026-07-15-r5/` admit zero pixels, labels, rows, models, or metrics. A different complete Ranjana manuscript is still required at the r5 boundary. The r6 BDRC sweep resolves that complete-source discovery gap without claiming a dataset. BDRC physical instance `MW1NPL132`, *Ārya nāmasaṁgīti*, is marked only `ScriptRanj` with print method Manuscript; linked scan `W1NPL132` is `AccessOpen` and `StatusReleased`, and image group `I1NPL132` exposes 21 unique IIIF canvases/services. The manifest SHA-256 is `a3c49a3fd68f30f0b4610a2f0c3d6432fd189caae5c371a0d482ae97fb02d11d`. BDRC's official access policy classifies scanned original manuscripts as public domain and permits sharing/adaptation for any purpose. The manifest itself has no license field; the work-record CC0 marker applies to metadata and is explicitly not treated as a pixel license. Five bounded 1200-pixel probes decoded: canvases 1 and 21 are cover/note material, while 5, 10, and 15 show dense visually Ranjana-like running text. Those observations are not qualified script decisions or a line-count audit. The frozen catalog at `outputs/newari-bdrc-ranjana-source-catalog-2026-07-15-r6/` records 21 exact remote resources, zero admitted pixels, zero audited lines, zero labels, and a prediction-free `omac` handoff. All pages remain one manuscript group; a distinct manuscript is still required for a manuscript-disjoint train/eval split. The r7 remote-acquisition round executed that BDRC handoff on `omacs-MacBook-Pro` (68,719,476,736 memory bytes), without OCR, training, or scoring. All 21 full-resolution JPEGs were fetched, SHA-256 recorded, and decoded at their exact manifest dimensions (all width 1200, heights 1676–1769); the capture totals 6,680,141 bytes and has 21 unique byte hashes. The source pixels remain on `omac`; the orchestration Mac mirrors only the metadata, hash list, dimension list, and rights/manifest evidence. A dynamic WordPress detail was caught rather than waived: BDRC's policy HTML randomizes CSS class-number suffixes across hosts, so raw page hashes differ while a narrow normalization of only those suffixes yields the same SHA-256 `99977ba3f0cb00da828a3833e6649a39c89c9f8e928a7127eeac1c35cd48d60e` and the required rights sentences remain present. The local adversarial audit at `outputs/newari-bdrc-ranjana-remote-acquisition-audit-2026-07-15-r7/` verifies the returned metadata bindings and creates a 21-row blank review worksheet. It explicitly cannot recompute the remote image hashes without the pixels. Qualified page decisions remain 0/21, audited lines remain zero, and no label, admission, model, or metric exists. The r9/r10 remote-acquisition round executed the independent Wellcome Bhujimol handoff on the same approved host, again without OCR, training, or scoring. All 217 requested review JPEGs were fetched, SHA-256 recorded, and decoded; the capture totals 58,471,370 bytes and has 217 unique byte hashes. Wellcome's `full/2000,/...` IIIF derivative decoded at width 2000 for 214 canvases and width 2001 for canvases 214, 216, and 217. The runner failed closed on the first 2001-pixel result, then the replay narrowed the accepted contract to exactly those observed widths and froze every exact dimension. The source pixels remain on `omac`; the local mirror is metadata-only. The adversarial audit at `outputs/newari-wellcome-bhujimol-remote-acquisition-audit-2026-07-15-r10/` verifies the returned bindings and creates a 217-row blank qualified-review worksheet. Qualified page decisions remain 0/217, audited lines and labels remain zero, and the entire manuscript is still one indivisible split group. A second Bhujimol manuscript is required for manuscript-disjoint evidence. The r12/r13 second-source round catalogs a distinct manuscript group for both Bhujimol and Ranjana without admitting data. The exact Wikimedia Commons JPEG of Cambridge `MS Add.2137` is photographer-authored and CC BY-SA 4.0. It contains four folio photographs; the first three selected regions are visually crosswalked to Cambridge canvases 48–50 (`28r`, `28v`, `29r`). Cambridge's official manifest independently calls all three hands hooked Nepālākṣarā (Bhujimol), but the complete 228-canvas institutional scan is CC BY-NC 4.0 and remains pixel-excluded. The single licensed composite is a separate manuscript group from Wellcome `MS Indic epsilon 1`, but cannot by itself establish the 200-line gate and still needs exact-original SHA-256 and qualified line review. BDRC `MW1NPL23` is separate from `MW1NPL132`; its open/released scan has 25 unique IIIF canvases under BDRC's scanned-manuscript public-domain policy. Its physical record assigns both `ScriptNewa` and `ScriptRanj` without page-level attribution, so every text-bearing canvas must be classified Ranjana/Newa/mixed/other by a qualified reviewer. The frozen r12 catalog stores no source pixels and reports zero qualified decisions, audited lines, labels, OCR, training, or scoring. The r13 inventory audit passes with 11 candidates (5 Bhujimol, 6 Ranjana), 3 acquisition-ready manifests, 2 completed remote acquisitions, and zero admitted sources. Its tests reject license, institutional-script, manifest-license, duplicate-service, probe-hash, and line/admission drift. Evidence: `outputs/newari-second-manuscript-source-catalog-2026-07-15-r12/` and `outputs/newari-source-candidate-audit-2026-07-15-r13/`. The r14/r15 acquisition round executed the second-source handoff on `omacs-MacBook-Pro` (68,719,476,736 memory bytes) without OCR, training, or scoring. The exact Commons original and all 25 BDRC canvases decoded at their frozen dimensions; all 26 byte hashes are unique and total 12,738,550 bytes (3,435,984 Bhujimol; 9,302,566 Ranjana). The Commons SHA-256 is `938013af2c4c16ac84164a41f903cf34d131663b7a069e58df779bacfcd4258d` and its upstream SHA-1 also matches. Separate profile checksum manifests bind one Bhujimol image and 25 Ranjana images, while the orchestration host mirrors metadata only. The local audit creates 28 blank qualified-review units: three Bhujimol panels (`28r`, `28v`, `29r`) and 25 BDRC canvases. All decisions, audited lines, labels, admissions, and claims remain zero. Evidence: `outputs/newari-second-source-prediction-free-review-2026-07-15-r14/` and `outputs/newari-second-source-remote-acquisition-audit-2026-07-15-r15/`. The refreshed r16 central inventory now reports 11 candidates, 7 explicit commercial-permission statements, 3 acquisition-ready manifests, 4 remotely fetched-and-hash-recorded sources, and still zero admitted sources or support claims (`outputs/newari-source-candidate-audit-2026-07-15-r16/`). The r17 review-readiness round joins the four acquired manuscript groups without copying pixels or inventing decisions. It binds 264 unique remote source images to 266 blank review units: Bhujimol has 218 images/220 units across Wellcome `b14901560` and Cambridge `MS Add.2137`; Ranjana has 46 images/units across BDRC `MW1NPL132` and mixed-script `MW1NPL23`. Both profiles meet only the two-manuscript-group precondition. Qualified decisions, estimated lines, audited lines, labels, admissions, OCR, training, scoring, and claims remain zero, so the >=200-line gate is explicitly false for both. Evidence: `outputs/newari-acquired-source-review-readiness-audit-2026-07-15-r17/` and `docs/handoffs/HANDOFF-newari-acquired-source-review.md`. The r43-r45 correction separates manuscript identity from exact transcription. The Wellcome story identifies `b14901560` / MS Indic epsilon 1 as the *Aṣṭasāhasrikā Prajñāpāramitā Sūtra*, but its 217 IIIF canvas labels are all `-`; no folio or line transcription accompanies the images. A GRETIL e-text exists for the Vaidya 1960 edition, but its page says it is for reference only, defers terms to the source file, and supplies no crosswalk to this manuscript. It is therefore neither rights-cleared prefill nor ground truth. BDRC's frozen physical record labels `MW1NPL99` “Unidentified” and supplies no e-text. Cambridge's official TEI says the *Nāradasmṛti* begins at verse 5.8cd on `28r`, but the manuscript interleaves that Sanskrit root with a Newari commentary-cum-translation; the TEI explicitly states that the Newari legal text and translation are unpublished and gives only short catalog excerpts. Accordingly, the r45 fail-closed audit reports three routes checked, zero exact line-aligned transcriptions, zero admissible prefills, and zero labels. The same pass found that the frozen r15 worksheet named `panel-28r`, `panel-28v`, and `panel-29r` but pointed all three rows at the same uncropped 2,332 x 1,800 four-folio Commons composite. The row identities and licensed source hash remain valid, so no frozen review data was rewritten. The r43 profile dispatch and r44 combined UI instead render fixed source-bounded regions `[0,0,2332,400]`, `[0,430,2332,420]`, and `[0,890,2332,410]`, crosschecked against institutional canvases 48–50. They fail closed to the full original if dimensions or bounds drift, preserve the full-image link and SHA-256, contain no OCR text or labels, and keep the same 339-row assignment scope. The complete selected regions were visually checked against the CC BY-SA Commons original. Tests cover rendering metadata, source-dimension refusal, immutable review scopes, output checksums, and the companion-text admission gate. Artifacts: `outputs/newari-qualified-page-review-dispatch-2026-07-15-r43/`, `outputs/newari-qualified-page-review-ui-2026-07-15-r44/`, and `outputs/newari-bhujimol-companion-text-audit-2026-07-15-r45/`. The r41 Devanagari-Newari source round found one exact official State Portal record titled `SIKKIM HERALD-NEWA EDITION`, dated 13 April 2022, under the permission-covered `sikkim.gov.in/uploads/SikkimHerald/` prefix. The existing Sikkim source policy remains exact at SHA-256 `2247033ac1ce7de8d99cad22feb702f9fe48a11341a2fb2d20c1d6d48b06717a`. An eight-byte HTTP Range probe confirms a PDF response with a total size of 3,975,535 bytes; no PDF body or page image was downloaded locally. The newer IPR portal currently returns zero Newar records and was used only as negative discovery evidence. The administration URL currently exposes only the official Government of Sikkim site shell, so no substantive publication claim is derived from it. The target writing-system route is Devanagari, but the edition's actual source script, fonts, and third-party content remain explicitly unconfirmed until all pages are rendered and reviewed on an approved remote host. The catalog therefore admits zero pixels, decisions, labels, real lines, models, or metrics. Evidence: `outputs/newari-devanagari-sikkim-source-catalog-2026-07-15-r41/`. Two tempting Commons catalog leads were rejected for production use in the same round. `AN-PM-12.pdf` and `AN-PM-09-Dvadasa Tirtha Yagu Bakham.pdf` are described on Commons as old Newari material in Devanagari with Public Domain Mark labels, but the British Library EAP record for AN-PM-09 states that access is for research purposes only. A downstream Commons rights label is not treated as authority to erase that source restriction. Recent uploader-applied Internet Archive CC0 metadata was likewise kept out when uploader authority could not be established. Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded 2026-06-18 scorecard in `docs/RECOGNIZER_GOAL_PROGRESS.md`): - **Limbu/Sirijonga 0.041 real-print** (champion: `runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23`, cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval). - **8 Devanagari languages 0.075–0.129 real-print** (Majhi, Magar, Sunuwar-Deva, Magahi, Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer. - **Sunuwar (Jenticha) 0.121 real-crop** after the same-font retrain (0.342 synth-only). - **Tirhuta 0.343 and Newa 0.613 synthetic-only** — weak, transliteration-fed, honest fails; Newa needs real printed-register Nepal Lipi, not more pixels. - **Kirat Rai (70 real crops) and Lepcha (132 real crops)** blocked on legacy-font converter coverage outside their eval editions. **Gurung Khema now has official real-print source material:** two 2022 State Portal Herald editions yield 749 mechanically convertible line candidates, but zero rows are admitted pending independent provenance review, native pixel-label review, and split/leakage gates. The near-term pack claim should therefore be script-family shaped: one Sirijonga recognizer, one shared Devanagari recognizer covering the many Devanagari-written languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa, Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval crops exist. **Focus order (2026-07-04 steer): Limbu first, Newar second.** The 2026-07-15 Newari steer requires Newar/Prachalit, Bhujimol, Ranjana, and Devanagari as separate writing-system profiles. Newar state: the Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own synthetic eval; transliteration-fed training failed; real printed-register Nepal Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20 Gorkhapatra pages; `newar` added to the registry (Devanagari primary + Prachalit secondary; audit passes); a frozen by-date pack built at `outputs/newar-devanagari-silver-pack-v1/` (train 4,096 / dev 1,528 / test 1,139 lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently verified; pack sha256 `1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85`); and the **first real-print Devanagari-Newar baseline** was scored locally (CPU export route, `tools/score_recognizer_pack.py`, per `docs/LOCAL_RECOGNIZER_SCORING.md`): the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at **codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158** (n=1,139 silver lines — machine-converted labels, diagnostic, not claim-grade). The same-day warm-start fine-tune (**newar-ft-v1**: deva-v2 base, LR 2e-5 cosine, 60 epochs, batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance `ashish-experiments`, ~68 min; artifacts + train log at `outputs/aws-newar-deva-ft-v1/`) scores the same frozen test at **codepoint CER 0.0465, grapheme CER 0.0955, exact-line 0.245** — a 25/27% relative CER drop and +55% exact-line over the deva-v2 baseline, scored through the identical local CPU path. **Newar's grapheme CER now sits inside deva-v2's own-language range (0.075–0.129): the second focus language reached Devanagari-pack parity from zero in one day.** Same claim boundary: silver machine-converted labels, diagnostic, not claim-grade until native review. Next Newar lever: native Prachalit (real printed-register data), and page-level Newar eval for the region-filtering track. ## Source Rule For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata only. Use `https://gorkhapatraonline.com/publications/naya-nepal` to find the language, date, and article, but do not use the blog/card image for OCR, training, or evaluation when the full paper exists. The authoritative source is the matching Gorkhapatra epaper PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus the text layer extracted from that same PDF page, with PDF URL/hash, page number, rendered image hash, extraction command/version, font evidence, and converter/map hash recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with `src/ocrtech/limbu_legacy.py` + `data/mappings/limbu-legacy/Limbu.map`. The same pattern — classify the text layer, then build an explicit converter only when the font requires it — replicates to every other language. Manual transcription is the fallback for no usable text layer, not the default. ## Getting More Gorkhapatra Pages (the specified harvest recipe) All numbers below were verified live on 2026-07-04. **Archive size.** The Naya Nepal publication archive paginates at `https://gorkhapatraonline.com/publications/naya-nepal?page=N` with 5 articles per page and runs to page ~352 before drying up: **~1,750 language-page articles total** (about 2 per day going back years). The 60-page crawl already yielded 300 items across 53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly older dates. **Epaper endpoint behavior (the critical constraint).** The date-range endpoint `https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra` **caps results at 30 PDFs per query and ignores `page=N` pagination**. A wide window silently drops most dates — the 2026-07-04 six-window discovery pass aligned only 111/300 items for exactly this reason, with unmatched dates interleaved among matched ones. **Seven-day windows return complete daily coverage (7/7 verified).** Always harvest with weekly windows. **The recipe** (one command; add one `--epaper-url` per week to cover the publication date span being harvested): ```bash env PYTHONPATH=src uv run --python 3.11 --with pymupdf \ ocrtech audit-gorkhapatra-language-pages \ --out outputs/gorkhapatra-language-pages-full- \ --publication-url https://gorkhapatraonline.com/publications/naya-nepal \ --publication-pages 80 \ --epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \ --epaper-url '...one per week, going back...' \ --download-pdfs \ --render-pages \ --timeout-seconds 180 ``` The command aligns every publication card (all languages at once — alignment is language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL), searches pages for the language-page markers (`भाषा पृष्ठ` and its legacy-encoding form `efiff k[i7`), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with **400 publication items, 195 aligned dates, 394 rendered language-page hits, zero warnings** at `outputs/gorkhapatra-language-pages-full-2026-07-04/`. Batch 2 (2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more items: **798 items, 294 aligned dates, 589 further page hits**, at `outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/`. The merged normalized inventory (`tools/build_gorkhapatra_language_pack_inventory.py`) shows **56 languages, 983 rendered language-page hits, most languages with ~20–27 pages spanning 14 months**, at `outputs/language-pack-inventory-merged-2026-07-04/`. For comparison, the entire previous state of the project had 8 usable pages for one language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least March 2024 — verified the epaper endpoint serves complete weeks there) is another ~2× and stays month-batchable with the same recipe. **Scaling to the full archive.** Full depth is ~700+ unique dates ≈ 35 GB of PDFs. Do not do it in one run: batch by month (4–5 weekly windows + the matching publication-page depth), run `extract-gorkhapatra-language-page-pdf-text` per batch, then keep rendered pages, extracted text, and hashes; PDFs can be pruned after extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume (`cdjk@100.110.148.43:/Volumes/TRANSCEND`) is the right home for a full-depth mirror. **Per-language downstream.** After harvest, run per language: ```bash env PYTHONPATH=src uv run --python 3.11 --with pymupdf \ ocrtech extract-gorkhapatra-language-page-pdf-text \ \ --candidate-language