prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
101 kB

Research State: Nepal Language Pack OCR

Updated: 2026-09-24

Devanagari recognizer fine-tune on k2 (2026-09-24)

Goal (owner steer): make the Devanagari route of the live demo (ampixa.thapa-ashish.com.np, k2 OCR worker) read citizenship-certificate-style and general documents properly. Resource inventory: docs/DEVANAGARI_RESOURCES_INVENTORY.md.

  • Base: official devanagari_PP-OCRv5_mobile_rec (568-entry dictionary incl. all Latin letters and digits); replaced our deva-v2 on the demo the same morning.

  • Training path: Paddle CPU training on k2 (Apple M2 Ultra) runs at 2 img/s, so the same network is trained with the transformers PyTorch port on MPS (70-90 img/s), exported to ONNX, and served through PaddleX create_model(engine="onnxruntime"). Code: tools/deva_rec_torch/. Preprocessing is bit-matched to PaddleX (98.8% line parity).

  • Data: real Gorkhapatra print re-cut into PDF-aligned chunks (tools/build_gorkhapatra_devanagari_chunks.py, edition-disjoint: test 2082-03-24, 2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11); synthetic citizenship/office/real-text lines in 44 OFL font families (tools/render_devanagari_form_synth.py); NVIDIA OCR-Synthetic-Multilingual-v1 English (CC BY 4.0) plus Liberation/DejaVu/Lato English renders.

  • Converter bug found and fixed: npttf2utf 0.3.7 mis-decodes the nayanepal alternate क्र ligature j|m (-> व्र, and प -> फ / stray m). 3,426 of 114,452 spans (3%) were affected (कार्यक्रम, प्रक्रिया, शुक्रबार, ...). Fixed in the chunk builder (RAW_FIXES); labels of all earlier Gorkhapatra-derived packs (incl. data/gorkhapatra-devanagari-real, the Newar and language-pack silver packs built from the same converter) carry the bug. Upstream fix still needed in nepal-ttf2utf.

  • Results, corrected labels, served path (PaddleX), codepoint CER / exact line (~/ocr-deva-ft/preds/final_comparison_v3.json on k2):

    Set official v2 v3 (deployed 13:07)
    E1v4 real chunks, held-out editions (n=3000) 5.13% / 0.47 0.69% / 0.91 0.57% / 0.93
    E7 real chunks containing क्र (n=530) 5.52% / 0.33 4.28% / 0.18 0.86% / 0.89
    E1b real full lines, independent labels (n=1500) 7.69% / 0.13 2.62% / 0.53 2.57% / 0.54
    E2 Newar test (n=1139) 9.11% / 0.06 3.53% / 0.35 3.54% / 0.35
    E3 synthetic citizenship-ne, unseen fonts (n=800) 6.78% / 0.56 0.84% / 0.93 0.75% / 0.94
    E3 synthetic office-ne, unseen fonts (n=300) 7.77% / 0.54 0.62% / 0.94 0.92% / 0.92
    E3 synthetic citizenship-en (n=300) 1.94% / 0.83 0.53% / 0.96 0.64% / 0.95
    E4 English, unseen Noto Sans (n=120) 2.57% / 0.62 1.58% / 0.73 1.64% / 0.72
    E4 mixed Nepali/English (n=120) 6.78% / 0.36 2.88% / 0.68 2.67% / 0.70
    E6 NVIDIA English, held-out pages (n=419) 10.6% / 0.71 2.99% / 0.84 2.84% / 0.84
    E5 handwriting probe (n=450) 64.7% 60.5% 61.0% (unusable)

    v2 learned the converter's क्र->व्र error (402 of 530 E7 predictions contained व्र); v3, a 6,198-step continuation on the corrected chunks-v4, removes it (2 of 530, same as the official model). Deployed as ~/ampixa-demo/models/ocr/space/models/devanagari-ft-v3; v2 and the official model remain on disk for rollback. E3 is synthetic from the same generator as training (different seed, held-out fonts) and overstates real-certificate accuracy; no real citizenship certificate has been tested. Real labels are PDF-text silver, not human-reviewed. Page OCR at 144 dpi garbles small text; at 288 dpi it reads cleanly.

  • Next: real citizenship samples (with consent) for a true eval; handwriting data (none licensed yet); upstream the j|m fix into nepal-ttf2utf and rebuild the other Gorkhapatra-derived packs; the demo's "Rows LTR" reading order interleaves newspaper columns.

Page pipeline on real government letters (2026-09-24, smoke test)

Two public notices from ocr.gov.np (a scanned Lalitpur District Court letter tilted 2°, and a Department of Industry letter), rendered at 200 dpi; 43 printed lines transcribed by hand (tools/deva_rec_torch/page_probe/: GT, probe scripts, results; PDFs on k2 `/ocr-deva-ft/layout-probe/; not a frozen pack, not claim-grade). Same detector config and v3 recognizer as the k2 worker; micro CER via tools/eval_metrics.py`:

Crop of each detected line cp CER grapheme CER exact
worker today: axis-aligned bbox of the quad + 12 px 21.4% 31.7% 0.40
rotated min-area quad (PaddleX CropByPolys), no pad 3.6% 5.5% 0.49
rotated quad + 6 px 2.9% 4.2% 0.65
rotated quad + 12 px 8.1% 13.3% 0.42
  • The worker's crop, not the recognizer, is the dominant error on real letters. _box_from_polygon flattens the detector quad to an upright box; on a 2° tilt a full-width line spans ~40 px vertically, so the crop swallows neighbouring lines (court-letter body lines read at confidence 0.24-0.57). Upstream PaddleOCR uses rotated crops; our port dropped them. Line-level evals (E1-E7) cannot see this. Not yet deployed: needs a page-level eval.
  • PP-DocLayoutV3 on the same pages: QR codes found (image, 0.95/0.96); emblems found (header_image); 0 of 2 stamps labelled seal (court seal -> header_image 0.65; the Industry stamp printed over the title not detected at all); signatures merged into the printed-name text block (0.52); no class for handwritten fill-ins; several printed lines (subject, dates, reference numbers) not boxed at the default threshold. Use layout to mask non-text regions, never to decide what gets OCR'd.
  • Signature/stamp crops produce junk text at recognizer confidence 0.35-0.71; damaged but real lines (tick mark, strike-through) score 0.74-0.82, so confidence can flag but not drop.
  • train.py now supports --guard-dev (print/synthetic/English dev sets may not regress more than --guard-tol over step 0); the plain dev mean let a 5-point handwriting gain hide a 1-point print loss. run_train_v4.sh on k2 uses it.

Paper-rewrite verification pass (2026-07-19): corrections to this document

While rewriting paper/DRAFT.md to cover the language pack, every number was re-verified against on-disk artifacts. Four values recorded earlier in this file do NOT match any surviving local artifact and must not be cited:

  • Newar "both-sides-normalized cp CER 0.0441" — no local artifact; the omac rescore output was never mirrored. The surviving verified score is 0.04655 cp / 0.09569 grapheme / 0.2450 exact (artifact-bearing labels, outputs/newar-ft-v1-verify-omac-2026-07-07/metrics.json, n=1,139).
  • Ol Chiki "public headline cp CER 0.0961 / n=358" — no local score artifact exists for the 358-row public-issue subset. Locally verified: v2 on the 772-line v3 pack scores 0.08264 cp / 0.2034 exact (outputs/olchiki-v2-run-2026-07-07/score-real-print-v3pack/metrics.json); the documented public-issue historical diagnostic remains 0.1018 (n=366) per docs/DATA_LICENSING.md. The v3 358-row candidate was never scored.
  • "299,611 per-language-attributed silver lines" — refers to the lost pre-crash batches. The rebuilt batches on disk total 306,971 (122,882 + 184,089, outputs/silver-lines-rebuild-b{1,2}-2026-07-05.log).
  • Ol Chiki v3 FT payload "5,330 entries" — the payload's own assembly manifest says 5,307 total images (242 real train + 5,000 synth + 65 real dev).

Also noted: the 8-language Devanagari scorecard (grapheme CER 0.075–0.129, 2026-06-18) lost its backing artifacts in the crash; it is text-only history now and the paper deliberately does not report it.

Machine-death recovery (read this first, 2026-07-05)

The Mac died 2026-07-04; the repo now lives at /Volumes/TRANSCEND/recovery-backup-20260704-172213/github/llm/ocr-tech (symlinked from the old /Users/cdjk/github/llm/ocr-tech path). outputs/ and runs/ are lost; code, .git, docs, spaces/ model exports (sha256-verified, champion included), and most of data/ survived. Every outputs/…/runs/… path below therefore refers to a lost artifact unless docs/RECOVERY_LEDGER.md maps it to a rebuilt one — read that ledger for the full survived/partial/lost inventory, recovery routes, and the rebuild policy (rebuilds go to new dated dirs). Batch-1 re-harvest launched 2026-07-05. Blocked on user credentials: AWS SSO (training checkpoints, newar-ft-v1 weights) and HF auth (Space cross-check).

Overnight lane results (2026-07-16/17)

These lanes delivered mechanically filtered silver substrate, not claim-grade training or evaluation data. The counts below establish file and archive parity; they do not replace native label review, edition-isolated leakage checks, or a real-domain score.

The Sikkim pack at outputs/sikkim-silver-packs-2026-07-16/ contains 9,743 Sunuwar/Jenticha clean real-page silver lines and 3,699 Kirat Rai clean real-page silver lines. In each case the summary.json clean-line count, on-disk labels.txt line count, and non-AppleDouble .tgz image-member count are identical. Sampled Sunuwar labels contain Jenticha characters in U+11BC0..U+11BFF, and sampled Kirat Rai labels contain Kirat Rai characters in U+16D40..U+16D7F, without non-ASCII characters outside the expected target block. Each summary records the eval manifests it checked. None of those eval editions occurs in the discovered source list, so the recorded excluded_editions: [] is consistent rather than evidence of an unperformed check.

Lepcha takes a different route. The diagnosis at outputs/lepcha-zero-yield-diagnosis-2026-07-16/REPORT.md shows that the new edition bodies are raster content and that the live PDF text is ordinary header/punctuation material, so there is no converter alias or byte-map fix that can recover the body. The converter lane is therefore closed and Lepcha moves to a synth-first recognizer route, retaining the 132-crop real pack recorded in data/lepcha-real-eval-pack/README.json as a real evaluation anchor. That pack is still machine-labelled and human verification remains pending, so it is diagnostic rather than claim-grade.

The Gorkhapatra full-depth backfill now covers every month from March 2024 through July 2025. The figures come from each month's gorkhapatra-language-pages.json; silver-kept comes from the corresponding silver summary and was independently matched to the .tgz member count.

Month Items Aligned Hits Silver-kept
2024-03 1,776 62 120 27,668
2024-04 1,731 61 129 21,429
2024-05 1,666 62 128 24,612
2024-06 1,601 60 118 21,924
2024-07 1,538 61 120 28,301
2024-08 1,473 60 120 24,534
2024-09 1,408 60 122 32,804
2024-10 1,343 53 103 25,055
2024-11 1,278 52 106 30,788
2024-12 1,213 61 121 39,115
2025-01 1,148 62 124 40,526
2025-02 1,083 56 112 35,391
2025-03 1,018 59 117 35,482
2025-04 953 57 113 34,815
2025-05 888 59 118 36,878
2025-06 823 60 120 37,836
2025-07 798 32 62 18,428
Total 21,738 977 1,953 515,586

All monthly audit warning arrays are empty, all archive counts match their silver summaries, and no source PDF remains under any monthly output. This is a delivered backfill substrate, not a statement that its page-attributed silver labels are language-verified.

Detector remediation remains active. The scale-normalized tile build is staged/running on omac; the local evidence at outputs/det-remediation-2026-07-16/REPORT.md ends at the completed code, config, and handoff. The GPU gate remains closed until the returned build summary and tile manifest pass the recorded re-audit checklist.

Current Goal

The end goal is an OCR system covering the multiple languages of the Nepal pack, built out of Paddle, with Limbu and Newar (Nepal Bhasa) as the first two focus languages (user steer, 2026-07-04). The system should accept real book/page images or PDFs for any language in the pack and output editable Unicode text with page, line, crop, confidence, and provenance evidence preserved. Limbu stays the proven template — it has the strongest recognizer and the deepest harvest. Newar is the deliberate second: it has its own native script (Prachalit), real cultural demand, and is currently the weakest trained recognizer, so it exercises every part of the method that Limbu already proved. Every stage that was built Limbu-specific must generalize: harvest, silver-label extraction, recognizer training, script routing, page/book OCR, and eval. The product is the full pipeline: page detection, script-aware routing to per-script recognizers, target-region and furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and review artifacts for hard lines. YOLO line detection stays a diagnostic/probe; Paddle-first is the production story.

The Language Pack

Two inventories exist and they are not the same thing:

  • The registry languages/nepal-v1.json tracks 18 target languages (17 Nepal + English) with scripts, fonts, dictionaries, and corpus status.
  • The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found 50 normalized languages (53 raw labels) in 400 articles over ~7 months (see the harvest section). Labels include all registry languages plus Chamling Rai, Bahing, Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo, Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi, Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more. Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा, ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) — tools/build_gorkhapatra_language_pack_inventory.py owns that normalization and the label→registry mapping.

Newari / Nepal Bhasa writing-system boundary (2026-07-15)

Newari support is explicitly four-profile, not a single generic "Newa" bucket:

Profile Source encoding OCR transcription Current capability
Newar / Prachalit Unicode Newa, U+11400..U+1147F Native Newa Route, training candidate, diagnostic checkpoint, and synthetic runtime smoke registered; no claim-grade real-domain CER
Bhujimol / Bhujinmol No separate Unicode script; configured as a Newa-codepoint style profile Newa plus mandatory source-style metadata Route, two acquired manuscript groups, and reviewer reference registered; zero qualified decisions, admitted labels, model, or CER
Ranjana Unencoded in Unicode 17.0 Newa transcription plus mandatory source image and Ranjana metadata Image route, two acquired manuscript groups, and reference font registered; zero qualified decisions, admitted labels, model, or CER
Devanagari Unicode Devanagari, U+0900..U+097F plus U+A8E0..U+A8FF Native Devanagari Route, diagnostic checkpoint/runtime smoke, and one permission-covered official Newa-edition acquisition candidate registered; source script and real-line quality remain unaudited

Unicode's current Newa chapter and code chart define the encoded Newa route. WG2 document N4555 describes Bhujimol/Bhujinmol as graphically distinct and discusses possible plain-text unification with Newa; that does not justify visual-model reuse. Ranjana is still represented by a preliminary encoding proposal, so this project does not invent a Ranjana Unicode range.

The canonical runtime profiles live in src/ocrtech/newari_scripts.py. The registry and book-OCR pipeline declare four separate recognizer references and preserve the source encoding/output policy in run metadata. The frozen audit at outputs/newari-four-writing-system-support-audit-2026-07-15-r1/ passes all four structural bindings while reporting model_ready=0/4 and four_script_ocr_claim_ready=false. "Supported" here means explicit input, profile, routing, and output semantics; each profile still needs at least 200 license-clean, pixel-label-audited, leakage-controlled real lines and a frozen real-domain evaluation before delivery or quality claims.

The r2 license-first discovery audit records three Bhujimol and three Ranjana source candidates without admitting any pixels. The most promising Bhujimol item is Asa Archives item 1147: its Internet Archive metadata has an uploader-applied CC0 URL, 63 scanned pages, and 325 automatically detected hOCR line boxes. Those boxes are acquisition-scale metadata, not labels. Uploader authority, institutional accession, exact-file SHA-256, qualified Bhujimol review, pixel faithfulness, and transcription mapping all remain open. For Ranjana, the Smithsonian's Karandavyuha Sutra record specifically marks image FS-6314_06 CC0 and its IIIF metadata identifies Ranjana, but the manifest contains only one canvas; the 77 physical folios are not 77 downloadable pages. British Library EAP790/21/121 identifies 12 Bhujimol TIFFs but restricts access to research use, and EAP1296/2/217 exposes 13 Ranjana images under noncommercial terms. The Nepalbhasa Manuscripts in UK archive states that copies and transcriptions are openly accessible but supplies no explicit reuse grant. The frozen audit is outputs/newari-source-candidate-audit-2026-07-15-r2/; its pass means the catalogue is fail-closed, not that any source, model, or OCR claim is ready.

The r3 provenance crosswalk matches item 1147 to the uploader's Asa Kuthi catalog spreadsheet at worksheet aiStuidoOutput, row 1170: DPN 1147, Bhāsvati, NRN ज्योति - २७१, RN 2919, Sanskrit/Nepalbhasa, N.S. 528, and 33.3 × 4.1 cm. The spreadsheet SHA-256 is 8484dbe95358b83d7d78d5d1752fafac5feeecfc007ccf5d1fe8b6ea3ce9bffd. This disambiguates a separate row whose RN, rather than DPN, is 1147. It does not attest Bhujimol because the matched catalog script field is blank, and it does not prove that the Internet Archive uploader was authorized to apply CC0. The exact authorization and reuse questions are staged in docs/NEWARI_ASA_1147_RIGHTS_REQUEST.md but have not been sent. The r3 frozen audit therefore remains PASS_CATALOG_INVENTORIED_ZERO_DATA_ADMITTED with no review manifest ready.

The r4 alternative-source sweep found a materially stronger Bhujimol route in Wellcome Collection. The official manuscript record identifies MS Indic epsilon 1, and Wellcome's editorial account of the object explicitly says it uses Bhujimol. Its complete IIIF manifest has 217 unique canvases/image services, SHA-256 596296750114948b39da2252b5a7b7c0738287dadcf7aed32939bfc29980c6d4, and a Public Domain Mark statement that expressly permits commercial copying, modification, and distribution. Wellcome describes 208 leaves with only six illustrated. Five 2000-pixel-wide probes at canvases 1, 50, 100, 150, and 210 decoded successfully; canvas 1 was a cover/label view, while the other four showed dense multi-block palm-leaf text. This is not a script-expert or label audit. Because the current host is an 18GB Mac, the repository freezes only a metadata acquisition manifest and an omac handoff; it does not bulk-download the 217 canvases. The whole manuscript must remain on one side of any split, so a separate manuscript is still required for leakage-proof training.

The r5 Ranjana follow-up resolved Wellcome MS Indic delta 33 without inflating selected images into a complete manuscript. The official physical record is closed-store only and has no digital item; the predictable IIIF presentation URL returns 404. Four exact digital-image records identify Ranjana and carry CC BY 4.0, but a prediction-free 1200-pixel probe shows one two-leaf overview, one partial-leaf environmental photograph, and two close-up miniature crops whose content overlaps the overview. The records therefore provide zero independent complete pages, not four. Their titles say AD 1653 while one description says transcribed in 1635; both values are retained. The frozen catalogues at outputs/newari-wellcome-ranjana-source-catalog-2026-07-15-r5/ and outputs/newari-source-candidate-audit-2026-07-15-r5/ admit zero pixels, labels, rows, models, or metrics. A different complete Ranjana manuscript is still required at the r5 boundary.

The r6 BDRC sweep resolves that complete-source discovery gap without claiming a dataset. BDRC physical instance MW1NPL132, Ārya nāmasaṁgīti, is marked only ScriptRanj with print method Manuscript; linked scan W1NPL132 is AccessOpen and StatusReleased, and image group I1NPL132 exposes 21 unique IIIF canvases/services. The manifest SHA-256 is a3c49a3fd68f30f0b4610a2f0c3d6432fd189caae5c371a0d482ae97fb02d11d. BDRC's official access policy classifies scanned original manuscripts as public domain and permits sharing/adaptation for any purpose. The manifest itself has no license field; the work-record CC0 marker applies to metadata and is explicitly not treated as a pixel license. Five bounded 1200-pixel probes decoded: canvases 1 and 21 are cover/note material, while 5, 10, and 15 show dense visually Ranjana-like running text. Those observations are not qualified script decisions or a line-count audit. The frozen catalog at outputs/newari-bdrc-ranjana-source-catalog-2026-07-15-r6/ records 21 exact remote resources, zero admitted pixels, zero audited lines, zero labels, and a prediction-free omac handoff. All pages remain one manuscript group; a distinct manuscript is still required for a manuscript-disjoint train/eval split.

The r7 remote-acquisition round executed that BDRC handoff on omacs-MacBook-Pro (68,719,476,736 memory bytes), without OCR, training, or scoring. All 21 full-resolution JPEGs were fetched, SHA-256 recorded, and decoded at their exact manifest dimensions (all width 1200, heights 1676–1769); the capture totals 6,680,141 bytes and has 21 unique byte hashes. The source pixels remain on omac; the orchestration Mac mirrors only the metadata, hash list, dimension list, and rights/manifest evidence. A dynamic WordPress detail was caught rather than waived: BDRC's policy HTML randomizes CSS class-number suffixes across hosts, so raw page hashes differ while a narrow normalization of only those suffixes yields the same SHA-256 99977ba3f0cb00da828a3833e6649a39c89c9f8e928a7127eeac1c35cd48d60e and the required rights sentences remain present. The local adversarial audit at outputs/newari-bdrc-ranjana-remote-acquisition-audit-2026-07-15-r7/ verifies the returned metadata bindings and creates a 21-row blank review worksheet. It explicitly cannot recompute the remote image hashes without the pixels. Qualified page decisions remain 0/21, audited lines remain zero, and no label, admission, model, or metric exists.

The r9/r10 remote-acquisition round executed the independent Wellcome Bhujimol handoff on the same approved host, again without OCR, training, or scoring. All 217 requested review JPEGs were fetched, SHA-256 recorded, and decoded; the capture totals 58,471,370 bytes and has 217 unique byte hashes. Wellcome's full/2000,/... IIIF derivative decoded at width 2000 for 214 canvases and width 2001 for canvases 214, 216, and 217. The runner failed closed on the first 2001-pixel result, then the replay narrowed the accepted contract to exactly those observed widths and froze every exact dimension. The source pixels remain on omac; the local mirror is metadata-only. The adversarial audit at outputs/newari-wellcome-bhujimol-remote-acquisition-audit-2026-07-15-r10/ verifies the returned bindings and creates a 217-row blank qualified-review worksheet. Qualified page decisions remain 0/217, audited lines and labels remain zero, and the entire manuscript is still one indivisible split group. A second Bhujimol manuscript is required for manuscript-disjoint evidence.

The r12/r13 second-source round catalogs a distinct manuscript group for both Bhujimol and Ranjana without admitting data. The exact Wikimedia Commons JPEG of Cambridge MS Add.2137 is photographer-authored and CC BY-SA 4.0. It contains four folio photographs; the first three selected regions are visually crosswalked to Cambridge canvases 48–50 (28r, 28v, 29r). Cambridge's official manifest independently calls all three hands hooked Nepālākṣarā (Bhujimol), but the complete 228-canvas institutional scan is CC BY-NC 4.0 and remains pixel-excluded. The single licensed composite is a separate manuscript group from Wellcome MS Indic epsilon 1, but cannot by itself establish the 200-line gate and still needs exact-original SHA-256 and qualified line review. BDRC MW1NPL23 is separate from MW1NPL132; its open/released scan has 25 unique IIIF canvases under BDRC's scanned-manuscript public-domain policy. Its physical record assigns both ScriptNewa and ScriptRanj without page-level attribution, so every text-bearing canvas must be classified Ranjana/Newa/mixed/other by a qualified reviewer. The frozen r12 catalog stores no source pixels and reports zero qualified decisions, audited lines, labels, OCR, training, or scoring. The r13 inventory audit passes with 11 candidates (5 Bhujimol, 6 Ranjana), 3 acquisition-ready manifests, 2 completed remote acquisitions, and zero admitted sources. Its tests reject license, institutional-script, manifest-license, duplicate-service, probe-hash, and line/admission drift. Evidence: outputs/newari-second-manuscript-source-catalog-2026-07-15-r12/ and outputs/newari-source-candidate-audit-2026-07-15-r13/.

The r14/r15 acquisition round executed the second-source handoff on omacs-MacBook-Pro (68,719,476,736 memory bytes) without OCR, training, or scoring. The exact Commons original and all 25 BDRC canvases decoded at their frozen dimensions; all 26 byte hashes are unique and total 12,738,550 bytes (3,435,984 Bhujimol; 9,302,566 Ranjana). The Commons SHA-256 is 938013af2c4c16ac84164a41f903cf34d131663b7a069e58df779bacfcd4258d and its upstream SHA-1 also matches. Separate profile checksum manifests bind one Bhujimol image and 25 Ranjana images, while the orchestration host mirrors metadata only. The local audit creates 28 blank qualified-review units: three Bhujimol panels (28r, 28v, 29r) and 25 BDRC canvases. All decisions, audited lines, labels, admissions, and claims remain zero. Evidence: outputs/newari-second-source-prediction-free-review-2026-07-15-r14/ and outputs/newari-second-source-remote-acquisition-audit-2026-07-15-r15/. The refreshed r16 central inventory now reports 11 candidates, 7 explicit commercial-permission statements, 3 acquisition-ready manifests, 4 remotely fetched-and-hash-recorded sources, and still zero admitted sources or support claims (outputs/newari-source-candidate-audit-2026-07-15-r16/).

The r17 review-readiness round joins the four acquired manuscript groups without copying pixels or inventing decisions. It binds 264 unique remote source images to 266 blank review units: Bhujimol has 218 images/220 units across Wellcome b14901560 and Cambridge MS Add.2137; Ranjana has 46 images/units across BDRC MW1NPL132 and mixed-script MW1NPL23. Both profiles meet only the two-manuscript-group precondition. Qualified decisions, estimated lines, audited lines, labels, admissions, OCR, training, scoring, and claims remain zero, so the >=200-line gate is explicitly false for both. Evidence: outputs/newari-acquired-source-review-readiness-audit-2026-07-15-r17/ and docs/handoffs/HANDOFF-newari-acquired-source-review.md.

The r43-r45 correction separates manuscript identity from exact transcription. The Wellcome story identifies b14901560 / MS Indic epsilon 1 as the Aṣṭasāhasrikā Prajñāpāramitā Sūtra, but its 217 IIIF canvas labels are all -; no folio or line transcription accompanies the images. A GRETIL e-text exists for the Vaidya 1960 edition, but its page says it is for reference only, defers terms to the source file, and supplies no crosswalk to this manuscript. It is therefore neither rights-cleared prefill nor ground truth. BDRC's frozen physical record labels MW1NPL99 “Unidentified” and supplies no e-text. Cambridge's official TEI says the Nāradasmṛti begins at verse 5.8cd on 28r, but the manuscript interleaves that Sanskrit root with a Newari commentary-cum-translation; the TEI explicitly states that the Newari legal text and translation are unpublished and gives only short catalog excerpts. Accordingly, the r45 fail-closed audit reports three routes checked, zero exact line-aligned transcriptions, zero admissible prefills, and zero labels.

The same pass found that the frozen r15 worksheet named panel-28r, panel-28v, and panel-29r but pointed all three rows at the same uncropped 2,332 x 1,800 four-folio Commons composite. The row identities and licensed source hash remain valid, so no frozen review data was rewritten. The r43 profile dispatch and r44 combined UI instead render fixed source-bounded regions [0,0,2332,400], [0,430,2332,420], and [0,890,2332,410], crosschecked against institutional canvases 48–50. They fail closed to the full original if dimensions or bounds drift, preserve the full-image link and SHA-256, contain no OCR text or labels, and keep the same 339-row assignment scope. The complete selected regions were visually checked against the CC BY-SA Commons original. Tests cover rendering metadata, source-dimension refusal, immutable review scopes, output checksums, and the companion-text admission gate. Artifacts: outputs/newari-qualified-page-review-dispatch-2026-07-15-r43/, outputs/newari-qualified-page-review-ui-2026-07-15-r44/, and outputs/newari-bhujimol-companion-text-audit-2026-07-15-r45/.

The r41 Devanagari-Newari source round found one exact official State Portal record titled SIKKIM HERALD-NEWA EDITION, dated 13 April 2022, under the permission-covered sikkim.gov.in/uploads/SikkimHerald/ prefix. The existing Sikkim source policy remains exact at SHA-256 2247033ac1ce7de8d99cad22feb702f9fe48a11341a2fb2d20c1d6d48b06717a. An eight-byte HTTP Range probe confirms a PDF response with a total size of 3,975,535 bytes; no PDF body or page image was downloaded locally. The newer IPR portal currently returns zero Newar records and was used only as negative discovery evidence. The administration URL currently exposes only the official Government of Sikkim site shell, so no substantive publication claim is derived from it. The target writing-system route is Devanagari, but the edition's actual source script, fonts, and third-party content remain explicitly unconfirmed until all pages are rendered and reviewed on an approved remote host. The catalog therefore admits zero pixels, decisions, labels, real lines, models, or metrics. Evidence: outputs/newari-devanagari-sikkim-source-catalog-2026-07-15-r41/.

Two tempting Commons catalog leads were rejected for production use in the same round. AN-PM-12.pdf and AN-PM-09-Dvadasa Tirtha Yagu Bakham.pdf are described on Commons as old Newari material in Devanagari with Public Domain Mark labels, but the British Library EAP record for AN-PM-09 states that access is for research purposes only. A downstream Commons rights label is not treated as authority to erase that source restriction. Recent uploader-applied Internet Archive CC0 metadata was likewise kept out when uploader authority could not be established.

Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded 2026-06-18 scorecard in docs/RECOGNIZER_GOAL_PROGRESS.md):

  • Limbu/Sirijonga 0.041 real-print (champion: runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23, cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval).
  • 8 Devanagari languages 0.075–0.129 real-print (Majhi, Magar, Sunuwar-Deva, Magahi, Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer.
  • Sunuwar (Jenticha) 0.121 real-crop after the same-font retrain (0.342 synth-only).
  • Tirhuta 0.343 and Newa 0.613 synthetic-only — weak, transliteration-fed, honest fails; Newa needs real printed-register Nepal Lipi, not more pixels.
  • Kirat Rai (70 real crops) and Lepcha (132 real crops) blocked on legacy-font converter coverage outside their eval editions. Gurung Khema now has official real-print source material: two 2022 State Portal Herald editions yield 749 mechanically convertible line candidates, but zero rows are admitted pending independent provenance review, native pixel-label review, and split/leakage gates.

The near-term pack claim should therefore be script-family shaped: one Sirijonga recognizer, one shared Devanagari recognizer covering the many Devanagari-written languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa, Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval crops exist.

Focus order (2026-07-04 steer): Limbu first, Newar second. The 2026-07-15 Newari steer requires Newar/Prachalit, Bhujimol, Ranjana, and Devanagari as separate writing-system profiles. Newar state: the Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own synthetic eval; transliteration-fed training failed; real printed-register Nepal Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20 Gorkhapatra pages; newar added to the registry (Devanagari primary + Prachalit secondary; audit passes); a frozen by-date pack built at outputs/newar-devanagari-silver-pack-v1/ (train 4,096 / dev 1,528 / test 1,139 lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently verified; pack sha256 1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85); and the first real-print Devanagari-Newar baseline was scored locally (CPU export route, tools/score_recognizer_pack.py, per docs/LOCAL_RECOGNIZER_SCORING.md): the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158 (n=1,139 silver lines — machine-converted labels, diagnostic, not claim-grade). The same-day warm-start fine-tune (newar-ft-v1: deva-v2 base, LR 2e-5 cosine, 60 epochs, batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance ashish-experiments, ~68 min; artifacts + train log at outputs/aws-newar-deva-ft-v1/) scores the same frozen test at codepoint CER 0.0465, grapheme CER 0.0955, exact-line 0.245 — a 25/27% relative CER drop and +55% exact-line over the deva-v2 baseline, scored through the identical local CPU path. Newar's grapheme CER now sits inside deva-v2's own-language range (0.075–0.129): the second focus language reached Devanagari-pack parity from zero in one day. Same claim boundary: silver machine-converted labels, diagnostic, not claim-grade until native review. Next Newar lever: native Prachalit (real printed-register data), and page-level Newar eval for the region-filtering track.

Source Rule

For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata only. Use https://gorkhapatraonline.com/publications/naya-nepal to find the language, date, and article, but do not use the blog/card image for OCR, training, or evaluation when the full paper exists. The authoritative source is the matching Gorkhapatra epaper PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus the text layer extracted from that same PDF page, with PDF URL/hash, page number, rendered image hash, extraction command/version, font evidence, and converter/map hash recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with src/ocrtech/limbu_legacy.py + data/mappings/limbu-legacy/Limbu.map. The same pattern — classify the text layer, then build an explicit converter only when the font requires it — replicates to every other language. Manual transcription is the fallback for no usable text layer, not the default.

Getting More Gorkhapatra Pages (the specified harvest recipe)

All numbers below were verified live on 2026-07-04.

Archive size. The Naya Nepal publication archive paginates at https://gorkhapatraonline.com/publications/naya-nepal?page=N with 5 articles per page and runs to page 352 before drying up: **1,750 language-page articles total** (about 2 per day going back years). The 60-page crawl already yielded 300 items across 53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly older dates.

Epaper endpoint behavior (the critical constraint). The date-range endpoint https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra caps results at 30 PDFs per query and ignores page=N pagination. A wide window silently drops most dates — the 2026-07-04 six-window discovery pass aligned only 111/300 items for exactly this reason, with unmatched dates interleaved among matched ones. Seven-day windows return complete daily coverage (7/7 verified). Always harvest with weekly windows.

The recipe (one command; add one --epaper-url per week to cover the publication date span being harvested):

env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
  ocrtech audit-gorkhapatra-language-pages \
    --out outputs/gorkhapatra-language-pages-full-<date> \
    --publication-url https://gorkhapatraonline.com/publications/naya-nepal \
    --publication-pages 80 \
    --epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \
    --epaper-url '...one per week, going back...' \
    --download-pdfs \
    --render-pages \
    --timeout-seconds 180

The command aligns every publication card (all languages at once — alignment is language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL), searches pages for the language-page markers (भाषा पृष्ठ and its legacy-encoding form efiff k[i7), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with 400 publication items, 195 aligned dates, 394 rendered language-page hits, zero warnings at outputs/gorkhapatra-language-pages-full-2026-07-04/. Batch 2 (2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more items: 798 items, 294 aligned dates, 589 further page hits, at outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/. The merged normalized inventory (tools/build_gorkhapatra_language_pack_inventory.py) shows 56 languages, 983 rendered language-page hits, most languages with ~20–27 pages spanning 14 months, at outputs/language-pack-inventory-merged-2026-07-04/. For comparison, the entire previous state of the project had 8 usable pages for one language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least March 2024 — verified the epaper endpoint serves complete weeks there) is another ~2× and stays month-batchable with the same recipe.

Scaling to the full archive. Full depth is ~700+ unique dates ≈ 35 GB of PDFs. Do not do it in one run: batch by month (4–5 weekly windows + the matching publication-page depth), run extract-gorkhapatra-language-page-pdf-text per batch, then keep rendered pages, extracted text, and hashes; PDFs can be pruned after extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume (cdjk@100.110.148.43:/Volumes/TRANSCEND) is the right home for a full-depth mirror.

Per-language downstream. After harvest, run per language:

env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
  ocrtech extract-gorkhapatra-language-page-pdf-text \
    <pack-manifest.jsonl> \
    --candidate-language <label> \
    --target-font <font> \
    --out outputs/<language>-pdf-text-<date>

then classify each page's text layer (native Unicode / legacy font / Devanagari transcription / Latin / none) and only then decide converter vs manual transcription.

Font/text-layer facts already established (30-page, 15-language extraction at outputs/gorkhapatra-multilingual-language-pages-current-2026-06-24-pdf-text/):

  • The body font of non-Limbu language pages is nayanepal — a legacy 8-bit Preeti-family encoding, not Unicode (8,845 of 11,791 spans; the text layer reads k[i7 ;+of]hs where the page shows पृष्ठ संयोजक). The converter already exists: nepal_ttf2utf.convert_devanagari(font="nayanepal") in the sibling repo /Users/cdjk/github/llm/nepal-ttf2utf (Preeti base map + nayanepal extension glyphs + NFC, ~97–99% clean on real Gorkhapatra pages), already used by tools/build_gorkhapatra_devanagari_real.py for the 79k-row real Devanagari corpus. The remaining leverage is wiring it per language: run it over each harvested language page and emit per-language silver line packs with the same faithfulness gate (clean-Unicode allow-set, conversion_gap accounting, leakage checks) that the Devanagari pack builder already implements. The same sibling repo also hosts the limbu, kiratrai, lepcha, and sunuwar legacy converters.
  • Tibetan-script fonts (TibetanMachine, MonlamUniOuChan5, Qomolangma-*) appear on Mugal and related pages, but every such span has empty extracted text (verified 2026-07-04 across all 2026-06-24 samples) — Tibetan-script pages are visual-only, like the known Limbu 2082-11-12 date. They need OCR bootstrap or manual transcription; no converter is possible.
  • Gorkhapatra-*, Nagarik-*, TimesNewRoman*, MyriadPro-* are masthead/furniture fonts — useful as non-target filters for target-region selection.
  • NamdhinggoSILL appears only on Limbu dates (the existing Limbu path).

What We Built (Limbu template, still true)

The Gorkhapatra alignment path: publication-page discovery, epaper-date matching, PDF download/rendering, page-marker search, PDF-native span/font extraction, review sheets, reference templates, and dashboard visualization. The expanded Limbu harvest has 9 discovered dates, 8 usable PDF-text-layer pages, 2,441 line rows, 2,430 usable silver labels (outputs/limbu-gorkhapatra-pdf-backed-silver-2026-07-03-v1/).

The Limbu recognizer training/eval path: 48,507-row combined pack; real-heavy fine-tune champion at codepoint CER 0.011336 / grapheme 0.024736 / exact-line 0.729 on the frozen 255-line silver eval. Synthetic hardening lessons: targeted+Unicode packs regressed; v3 dictionary warm-start catastrophically failed (~0.98 CER); keep the real-heavy v2 champion; future gains come from real/book-domain harvesting or a deliberate rehead, not blind synth rows.

The page/book OCR loop: local + HF Space (spaces/limbu-ocr/, deployed private at https://huggingface.co/spaces/ampixa/limbu-ocr) with line OCR, Paddle Page OCR, Paddle Book OCR, image-line fallback, runtime checks, annotated pages, and crops.

Page-level diagnostics on the 8-page silver pack: Rows LTR + y/conf70 filter scores page cp CER 0.414330, line F1 0.963009, matched-line cp CER 0.026039, order accuracy 0.993176. Recognition on matched lines is strong; full-page CER is dominated by page reconstruction, target-region selection, and furniture filtering — that is the pipeline problem to solve, and it is language-independent.

Supervised line-detector infrastructure from CDC PDF geometry (287 pages, 26,160 annotations, mAP50 ≈ 0.921 on PDF-rendered val) — evidence the detector path works, not a solved product.

A claim-grade eval harness (versioned packs, hashes, provenance, normalization profiles, cp/grapheme CER, CIs, paired comparisons, leakage manifests) — parked as a guardrail; do not let it eat work blocks that should go to the pipeline.

Current Working State

The strongest component is the Sirijonga line recognizer. The weakest component is end-to-end page/book reconstruction: target-region selection, furniture filtering, line preservation, and ordering. Of the three Limbu-locked code spots, two were generalized on 2026-07-04 (Codex worker, tests green): _script_ratio in src/ocrtech/engines.py is now driven by the shared SCRIPT_CODEPOINT_RANGES table covering all pack scripts (devanagari, limbu/sirijonga, latin, sunuwar/jenticha, lepcha, kirat_rai, newa/prachalit, tirhuta, gurung_khema, ol_chiki, tibetan, with aliases), and the image-line filter in src/ocrtech/image_lines.py accepts any script key from that same table. The third spot fell the same day: the src/ocrtech/limbu_runtime.py preflight now takes required_script (default limbu, validated against the shared table; CLI --required-script), so a router card for any pack script can be preflighted. The engine classes themselves (CandidateEngine, PaddleOcrEngine, ScriptSelectCompositeEngine, build_image_line_document, structure.py reading order) are already parameterized — the rest of the generalization is registry-driven wiring, not a rewrite.

The Paddle Pipeline (whole pipeline, not just recognizers)

Target architecture, all stages Paddle-first and language-pack-parameterized by the per-language pipeline config configs/nepal-language-pack-pipeline-v1.json (50 languages: label, registry id, scripts, recognizer artifact, dictionary, converter, text-layer class, harvest counts — generated 2026-07-04 from the inventory):

  1. Capture/prep — existing prepare-limbu-capture path, already language-agnostic in behavior; rename/rescope.
  2. Detection — Paddle text detection as production default; YOLO line detector and morphology image-line boxes stay as diagnostics/rescue passes.
  3. Script identification & routing — generalize _script_ratio to a Unicode-block-driven scorer over the pack's script list (Devanagari, Limbu, Jenticha/Sunuwar, Lepcha, Kirat Rai, Newa, Tirhuta, Gurung Khema, Ol Chiki, Tibetan, Latin), then route each line to the script's recognizer from the registry.
  4. Recognition — per-script PP-OCRv5 recognizers (existing runs) with per-script dictionaries and input-width profiles (Sirijonga width-480 lesson).
  5. Target-region & furniture filtering — non-oracle region selection using layout plus font/language signals; furniture fonts and masthead geometry as negative evidence. This is the main open research item; score against the Rows LTR + y/conf70 baseline.
  6. Reading order & assembly — auto_layout selection (wide-column RTL vs rows) per page, conservative assembly stays diagnostic.
  7. Export — Unicode text, JSON with provenance, Markdown; review queues + dashboards per page (existing Limbu artifact shapes, renamed pack-wide).
  8. Eval guardrail — frozen real-page packs per language: page CER/WER, line P/R/F1, reading order, blank/extra/missed lines, review burden.

PP-StructureV3 and PaddleOCR-VL remain the documented fallback/teacher direction for hard layouts, per docs/NEPAL_MULTILINGUAL_OCR_GOAL.md.

Production book-ingestion plan (2026-07-04)

The production target: a book (scanned PDF, image folder, phone captures) in; editable Unicode, provenance JSON, and Markdown out; a review queue covering only lines that need a human; frozen regression packs gating every model/config change. The production KPI is review burden (can an editor fix a book in hours), not CER alone. Phases in leverage order:

  1. PP-StructureV3 spike — DONE 2026-07-05, verdict: stays fallback. The 2026-07-04 CPU run died producing nothing (~6 h on page 1/8); the re-run on the L4 GPU did all 16 rebuilt pages in ~4 s/page. Scored on the rebuilt 16-page / 4,895-line Limbu silver pack, same champion recognizer and same run_page_ocr_eval.py flags in every arm (diagnostic, non-claim):

    arm page cp CER det F1 matched cp CER order acc
    PP-StructureV3 raw (its own reading order) 0.797 0.904 0.0345 0.778
    PP-StructureV3 boxes + Rows-LTR + y/conf70 0.421 0.934 0.0331 0.993
    Current stack: Paddle det + Rows-LTR 0.437 0.942 0.0266 0.993
    Current stack + y/conf70 filter (baseline) 0.413 0.966 0.0266 0.993

    Stock PP-StructureV3 loses on every axis even after grafting our reading order and filter onto its boxes: worse detection, worse crops for our recognizer (matched CER 0.033 vs 0.027), far worse native reading order, and it needs a GPU. The Paddle det + Rows-LTR + y/conf filter chain stays the production backbone; PP-StructureV3 remains the documented fallback / teacher for table- and figure-heavy layouts (not exercised by these text-heavy newspaper pages). Artifacts: outputs/ppstructure-spike-rebuild-2026-07-05/ (stage-1 GPU JSONs, stage-2 documents, all score dirs); baseline outputs/limbu-baseline-rebuild-2026-07-05/.

  2. Detector fine-tune — cash in the supervised det goldmine: 983 language pages with PDF-text-layer line geometry across 50+ languages plus 26k CDC annotations; fine-tune PP-OCRv5 det to fix the center-column miss pattern.

  3. One config-driven book runner — generalize run_limbu_book_ocr.py into a language-pack run_book_ocr (input → prep → layout → det → script-route → recognize → assemble → export + review queue), resumable per page.

  4. Acceptance gates — frozen page packs (Limbu 8-page, Newar pages, one real book) with thresholds on page CER, line F1, reading order, review-burden %; the parked eval harness serves as the gate runner, not a research project.

  5. Packaging/serving — versioned per-script model bundles, Space/API on the pack pipeline, CPU eval path, GPU batch for book jobs.

External Native-Script Real-Data Harvest (2026-07-06)

Rebuilt the lost newa-zenodo-6967421-v1 asset and swept Zenodo/HuggingFace/ GitHub/Internet Archive for other real (non-synthetic) native-script OCR material. Everything landed at data/external-language-resources/native-script-real-2026-07-06/<script>/ with a provenance.json per source (license, byte size, md5, verify method). Full details in that tree; headline results:

  • Newa/Pracalit (Zenodo 6967421, CC-BY-4.0): re-downloaded, md5-verified (76d464a6b656482b7c0594c0cb37fb18), extracted. Correction to the record: it is 441 page images + 441 PAGE-XML, not the 683 pages this doc previously cited — 441 matches the record's own metadata.xml and the Zenodo description's "441 training pages"; the "242 validation pages" the same description mentions are not in this file. Do not carry 683 forward. Mirrored bit-for-bit at heiDATA (10.11588/DATA/WI9184, same title/size) — useful as a second-source integrity check, not new content. tools/newa_zenodo_build.py expects data/newa-zenodo/extracted/; this pull is at the new path above, so point the tool there (or symlink) before re-running the line-crop pipeline.
  • Newa/Pracalit, other leads that did NOT pan out: a newer O'Neill paper (Zenodo 17268040, "Revisiting HTR for Nepalese Manuscripts in Pracalit Script," 2025) cites no new public GT beyond 6967421. A published "First-Ever Prachalit Newa Lipi Character Dataset" paper exists, but its data (via sobit-nep/Prachalit-Script-Character-Recognition-using-Deep-CNN on GitHub) is a dead Google Drive link (404, not actually "anyone with the link") — verified, not assumed. heiDATA's FID4SA-GT dataverse (the same group that hosts the Newa mirror) also has Devanagari, Malayalam, and Bengali printed GT sets, CC-BY/CC0 — none of them our target scripts, not pulled.
  • Ol Chiki (Santali): no image+GT dataset exists on Zenodo/HF/GitHub (only a Tesseract .traineddata blob and app scaffolding with no bundled data). Internet Archive had real wins instead: chiki-chemed (a Santali primer, real scanned pages, CC BY-NC-ND 4.0 — note the NC-ND clause before any commercial use) and aale-chhatka-pdf-e-magazine-{2023,september-2023} (a real monthly Ol-Chiki literary e-zine, self-published by its own editor's archive.org account; the 2023 item carries the Creative Commons Public Domain Mark 1.0, which is neither CC0 nor a license, while the September item has no license field despite the same publisher — don't assume terms transfer). All three downloaded, size-verified, script-confirmed by rendering a sample page.
  • Tirhuta/Mithilakshar — best new lead of this pass: the Maithili e-zine Videha (world's first Maithili webzine) publishes a parallel Tirhuta edition of every issue, archived at Internet Archive under the videha_* family, CC BY-SA 4.0, issues 001–149 confirmed present (3 items, ~12 GB) plus larger combined items (videha-262 ~23 GB, videha-306 ~40 GB) referencing further issues — not bulk-downloaded, catalogued only. Pulled one sample PDF and confirmed by rendering it that the pages are genuine Tirhuta-shaped glyphs, but pdftotext on the embedded text layer yields Devanagari-range codepoints, not the true Tirhuta Unicode block (U+114xx). That means these are legacy glyph-substitution fonts — the same shape of problem this repo already solved for Gorkhapatra Devanagari (nayanepal/legacy Preeti-family). If a Videha-specific font map can be built, this is real running text at real periodical scale, the same playbook as the Limbu win. This is the single most promising unexploited source found this session.
  • Lepcha / Kirat Rai / Sunuwar / Limbu (Sikkim Herald): the four newer IPR listing endpoints (?language=Rai|Mukhia|Lepcha|Limboo) remain useful for discovery, and no Zenodo/HF/GitHub/IA periodical archive was found. This 2026-07-06 note is superseded for admission by the 2026-07-15 rights audit: the express reproduction permission covers material hosted on the official State Portal, not automatically ipr.sikkim.gov.in. The IPR tool is now metadata-only by default; admit only a mirrored sikkim.gov.in/uploads/SikkimHerald/ file or separately permitted IPR file.
  • Nepal Lipi periodicals: the Internet Archive nepalbhasatexts collection (477 items) is almost entirely Devanagari-script Nepal Bhasa print media (Situ, Jhee, Thaunkanhe, Khonapau, etc. — already the domain of the existing Newar-Devanagari track, not this one). Checked several candidates that looked native-script by title: kmc-notice-2023 is a single decorative Ranjana logo clipping, not running text. NepalLipiSangrahaByHemrajShakyavamsha_201712 (no stated license) is a Devanagari-language book about Nepal Lipi that does contain real alphabet-plate specimens of historic native scripts (Kunmol, etc., with transliteration) — a glyph reference, not periodical running text. No genuine Nepal-Lipi-script periodical PDF archive was found.

Recommendation, one line per script: Newa — nothing left to pull, move to building the line-crop pipeline on the corrected 441-page set. Ol Chiki — use aale-chhatka-pdf-e-magazine-2023 (publisher/uploader Public Domain Mark 1.0, confirmed script) as the first real training/eval seed. Tirhuta — decode the Videha legacy Tirhuta font (highest leverage find this session: real, licensed, periodical-scale) before anything else. Lepcha/Kirat Rai/Sunuwar/Limbu — use the IPR listing only to discover editions, then catalog hash-pinned mirrors on the permission-covered State Portal; do not bulk-download IPR PDF bodies without separate exact-prefix permission evidence.

Native-script wave 1 results (2026-07-06)

  • Ol Chiki v1 (first recognizer for the script here): PP-OCRv5 mobile from generic pretrained, 56,640 synth lines (Santali Wikipedia corpus × 3 Noto weights × 6 degradations), 25 epochs. Synth eval best exact-line 0.546 / norm_edit_dis 0.800. Real print (234 Aale Chhatka magazine lines via the new OLCKOptimum converter): cp CER 0.2245, exact-line 0.184 — real-print performance ≈ synth-eval performance, so the synthesis recipe transfers; quality itself is the v1 limit. Known levers, in leverage order: (1) same-font synth — we hold the extracted OLCKOptimum TTFs; replay the Sunuwar same-font lesson (0.342→0.121); (2) resolve the 2 uncertain converter bytes (n,T) to unlock 653 more real lines for train/eval; (3) real-line fine-tune on an issue-split of those lines. Artifacts: outputs/olchiki-v1-run-2026-07-06/ (checkpoint+export+scores), pack data/olchiki-real-eval-pack/, converter nepal-ttf2utf@acab167. Claim note: Santali OCR exists elsewhere (India); this is pack coverage, not a first-ever claim. v2 (same-font, 2026-07-07): re-rendered the same corpus with the OLCKOptimum fonts extracted from the magazine PDFs (all 48/48 block chars covered by the embedded subsets) + one Noto weight: real-print cp CER 0.1911, exact-line 0.321 (same 234-line pack) vs v1 0.2245 / 0.184 — the same-font lever works on this script too, though less dramatically than Sunuwar's; remaining error is heading/display type, tight-leading crop bleed, and genuine model limits. Artifacts: outputs/olchiki-v2-run-2026-07-07/. A later asset-independence audit found that those fonts came from the exact evaluation edition, so v2 remains a useful diagnostic but cannot provide an edition-independent claim result.
  • Newa native v1 (manuscript domain): honest failure. 150 epochs on the recovered Zenodo 441-page GT (2,391 train lines, by-manuscript split, 800px width, PP-OCRv5 mobile from generic print pretrained): dev exact 0.0016 / norm_edit 0.144, and crucially train-side norm_edit is also 0.146 — the model could not fit its own training set, so this is a recipe failure (print-pretrained mobile backbone vs dense 16th–19th-c. handwriting at 63–154 chars/line), not a hand-generalization failure. The reference point (O'Neill 2022/2025, Transkribus HTR on the same data) shows the task is learnable with HTR-specific training. v2 design: synth-first curriculum — render the TRAIN-split transcription text (plus the 36k-char web corpus; no test text, no transliteration) in the 2 Unicode Newa fonts with degradations, pretrain, then manuscript fine-tune; consider the server rec backbone. Artifacts: outputs/newa-native-v1-run-2026-07-07/.

Fan-out results (2026-07-13, Codex wave)

  • Ol Chiki uncertain bytes RESOLVED (nepal-ttf2utf@5822f30): bytes n→ᱱ (EN U+1C71) and T→ᱛ (AT U+1C5B), each confirmed by three independent evidence lines (glyph stroke features, corpus context distributions vs the 204k-line Wikipedia corpus, lexicon hit-rates 0.89 vs 0.57–0.60 runner-up); evidence at outputs/olchiki-uncertain-bytes-2026-07-13/. Clean real-print yield rose 234 → 895 lines; expanded pack at data/olchiki-real-eval-pack-v2/. olchiki-v2 on the 895-line pack: cp CER 0.1275, exact-line 0.197 (scored on omac, 5m18s) — the new body-text-dominated benchmark reads meaningfully better than the display-type-heavy 234-line pack suggested (0.1911). Pack v2 was initially treated as the reference eval, with 895 candidate lines. The following audit superseded that interpretation before any claim use. Corrections (2026-07-14, license audit + label audit): (a) public claims may use only the publisher/uploader public-domain-marked issue; the mark is not CC0 or a license; (b) the label audit found 67/895 crops blank (pack-builder bug, all in that issue). The resulting cp CER 0.1018 / exact 0.191 (n=366) and combined clean 0.0826 (n=828, internal) are historical diagnostics, not claim headlines: surviving crop/label mismatches, ambiguous punctuation, incomplete provenance, and same-edition font leakage remain. Also found: the Newar-Devanagari packs carry systematic converter whitespace artifacts (spurious space after halant / before combining marks; ≥1 in ~45% of test lines) — word content is faithful, but a whitespace-normalization pass is required before those numbers are claim-grade. See docs/DATA_LICENSING.md and the verification-wave section.

Newa root cause found (2026-07-15): CTC timestep cap, not data or height

The height-ablation prep (dry-runs on omac; artifacts at outputs/newa-height-ablation-prep-2026-07-14/, VERDICT.md) found the mechanical cause of the v1 training failure: PPLCNetV3's train-mode forward ends in adaptive_avg_pool2d([1,40]), hard-capping the CTC sequence at T=40 timesteps at any input height or width — and 95.7% of manuscript train lines exceed 40 chars (median 63, max 154). CTC cannot emit more labels than timesteps: v1 could never fit its own training set, and a pure height ablation is predicted to reproduce the failure. This coheres with every other result: Limbu/Ol Chiki/Newar-Devanagari lines mostly fit ≤40 chars and their models trained fine. Also verified: heights 96/128 train as pure config changes (3 finite loss steps each), but the EVAL path breaks (Im2Seq asserts H==1) — a verified 1-line patch ships in the prep dir; pretrained weights load identically at all heights; an earlier T=40 memory estimate proposed L4 batches 16 (h96) / 12 (h128), but T=200 requires a fresh approved-GPU measurement. The corrected Newa experiment therefore needs a single train/eval pool width of T=200 (the exact CTC lower-bound maximum is 176 after normalization), h96, and only the CC-BY-4.0 Zenodo manuscript source. A backward-compatible pool patch has now been shape-tested on omac: defaults remain train T=40 / eval T=100, while the opt-in h48 and h96 paths both produce T=200 and a length-162 CTC loss is finite. The pre-existing zero-byte newa_real_000041.png was not dropped: it was restored from the authoritative local master and pinned at SHA-256 c86f18e93b123a87e26682b973c5db8a2f8931434f11f6c88a63105161920360.

Native-script verification round (2026-07-15)

  • Newa CTC200 payload staged, not trained. Labels are narrowly normalized (NFC, NBSP to space, whitespace runs collapsed, boundaries trimmed), and every referenced crop decodes with ink. Exact-label and 16-codepoint substring isolation removed 430 of 2,391 training rows, leaving 1,961. Hitopadesa is the 144-line checkpoint-selection dev manuscript; the 635-line Cambridge manuscript is sealed as claim_eval and must not enter training or model selection. Post-filter overlap is zero and all labels are feasible at T=200. A prediction-free all-row Cambridge review bundle has now been built on omac from the exact payload and original Zenodo export: 635/635 bundled PNGs are byte-identical to their payload crops, all 635 decision rows remain blank, and the archive is explicitly not claim evidence. The crop labels still need documented native pixel review; any corrections require a fresh post-review exact/16-codepoint leakage audit before a claim-eligible freeze. A fail-closed freezer now revalidates the exact payload/source/review archives, immutable fields, pixels, all 635 decisions, normalization, and the >=200-row dedup gate. It keeps non-attested outputs claim-false and requires exact CTC200, train/dev, post-review leakage, checkpoint, and training-history bindings for claim readiness. The real blank worksheet fails atomically; no reviewed pack exists. GPU training remains approval-gated. Artifacts: outputs/newa-ctc200-prep-2026-07-15/, outputs/newa-cambridge-review-v1/, and docs/handoffs/HANDOFF-newa-ctc200.md.
  • Newa balanced sampling is now a deterministic, disabled candidate. The exact 1,961-row train-only capsule contains 146,153 Newa codepoint tokens across 69 types and 87,777 extended-grapheme tokens across 585 types. The schedule excludes the 2 codepoint and 71 grapheme singleton types, adds 490 references under a three-occurrence row cap, and preserves exactly the same 1,961 unique image/text pairs. Train/Hitopadesa exact-label and 16-codepoint overlap are zero before and after scheduling; no Hitopadesa or Cambridge row is copied. The six-file artifact reproduced byte-identically when rebuilt with the pinned regex==2026.5.9 segmentation runtime. This is a sampling heuristic, not accuracy evidence, and no current payload or config enables it. Activation requires a newly hash-bound training path, approved GPU ablation, and the final corrected-Cambridge leakage audit. Artifact: outputs/newa-balanced-sampling-v1/.
  • Stock RepSVTR is a runnable CTC200 fallback, not a trained result. A no-image, batch-1 CPU probe on omac used the clean PaddleOCR v3.7.0 checkout and the official hash-pinned RepSVTR checkpoint. At H96xW1600, the train-mode CTC branch and eval-mode full forward both emit T=200, leaving 24 timesteps above the audited corpus maximum of 176; an alternating length-176 synthetic CTC loss is finite. All 301 backbone tensors and 394/398 total tensors are pretrained-compatible, with only the four Newa-vocabulary-sized tensors reinitialized. A follow-up synthetic CPU probe then exercised the exact five-tensor loader contract through one full train-mode MultiHead forward, CTC+NRTR MultiLoss, backward, and scheduler-positioned AdamW update. It observed CTC [1,200,108], NRTR [1,177,112], finite component losses, 279/279 finite nonzero gradient tensors, and finite positive updates in the backbone plus both heads; peak RSS was 1,646,444,544 bytes. No image, label file, dataset row, prediction, score, GPU, cloud, or network action occurred. The exact configured warmup starts at LR 0, so the probe explicitly advanced the scheduler 1,961 times without optimizer calls before its sole update at LR 4e-5; this is mechanism evidence, not first-iteration chronology or training-loop history. Together the probes establish an independent backbone/temporal path from patched PPLCNetV3 and close the synthetic full-loss/backward/update gap, not the framework, real-data, GPU-memory, convergence, or accuracy gaps. Training remains approval-gated, and selecting this path requires the distinct stock-RepSVTR claim-attestation profile now implemented in tools/audit_newa_cambridge_claim_readiness.py and its freezer. That profile pins the exact config, feasibility report, fully clean PaddleOCR checkout, stock sources, official initialization, approval, run log, selection record, and a CPU-loaded final state dict with the exact 398 Newa model keys/shapes. The freezer independently requires the raw approval/run/log/selection records and the selected checkpoint. The legacy four-file PPLCNetV3 profile remains unchanged. Identity and execution authenticity are still an external custody/signature boundary; no real approval, training record, compatible final checkpoint, attestation, or score exists. Artifacts: outputs/newa-repsvtr-ctc200-feasibility-v1/ and outputs/newa-repsvtr-full-train-step-v1/.
  • Ol Chiki v2 withdrawn from claim use; v3 is only a review candidate. A full audit found blank/rule-only crops, unrelated or neighboring text, unresolved legacy punctuation, a stale converter-commit claim, and training fonts extracted from the evaluation edition. The public-source-only v3 builder pins the PDF and resolved map, uses neighbor-clamped vertical crops, rejects blank renders, emits line-level hashes/provenance, and refuses to call its output frozen or claim-eligible. The remote candidate contains 358 rows after 75 blank renders were excluded. Every row still requires review by an Ol Chiki reader; no new score is permitted before freeze and independent exact/substring and asset-isolation audits. A fail-closed freezer validates the pinned candidate and all-row review archives, the source PDFs, every crop and provenance binding, and all 358 immutable review decisions; it refuses claim eligibility without separately bound leakage and asset-isolation attestations. No decisions or frozen pack exist. Artifacts: outputs/olchiki-claim-eval-v3-2026-07-15/.
  • Tirhuta now has an edition-disjoint train/eval candidate route, not a score. PyMuPDF retained glyph IDs for all 3,306 U+FFFD characters in the 152-page issue-001 sample. A fail-closed 164-entry functional map recovered every one; 30,396/30,581 Janaki trace fragments then converted cleanly (99.395049%). Issue 001 now supplies 400 nonblank real-line eval candidates across 116 pages and a prediction-free all-row review packet. The separately dated, CC BY-SA 4.0 April 2008 issue required 34 additional GID mappings. Exhaustive shaping with the hash-pinned Janaki 1.000 reference font reproduced all 34 plus the prior 164 with zero regressions. Its 2,000-row candidate has 2,000 unique image hashes and zero image-audit issues; removing one 16-codepoint cross-issue substring conflict leaves 1,999 training rows with zero exact or long-substring overlap. Thirty new GIDs have clean crop witnesses and four have shape evidence only. The 400 eval rows and 30 focused mapping rows remain native-review pending; no model or CER exists. Artifacts: outputs/tirhuta-videha-gid-recovery-v1/, outputs/tirhuta-videha-line-candidate-v1/, and outputs/tirhuta-videha-april2008-train-candidate-v1/. A separate fail-closed freezer now validates the two pinned issue-001 archives, source PDF, packaged audits, converter/runtime provenance, and the exact crop bytes shown to reviewers. It requires all 400 immutable decisions, at least 200 nonblank unique accepted lines, and Tirhuta-bearing human labels. Claim eligibility remains false unless an independent attestation binds the reviewed label-file hash and the exact April train label/manifest hashes and passes exact-text, 16-codepoint substring, edition, and model-selection isolation. The decisions and attestation do not yet exist, so no pack was frozen and no score was run.
  • The other-script registry was downgraded to extant evidence. The Kirat Rai pack has 70 rows / 53 unique labels, and all 53 occur in its real-render training inventory. The shared Devanagari harvest has 79,364 labels but only 40,358 images, so it is a training inventory with 39,006 missing assets, not eval. Sunuwar's historical 429-row pack/checkpoint and Lepcha's source PDFs and checkpoint are absent. Historical metrics remain historical and none of these routes is a current script delivery. Registry evidence: configs/nepal-language-pack-pipeline-v1.json, configs/recognizer-eval-pack-normalization-v1.json, and configs/eval-harness-recipes-v1.json.
  • Sikkim Herald source permission was found and the three routes reopened. The earlier Indian-government-work term analysis missed the Government of Sikkim State Portal's express copyright permission. Its official copyright page permits free reproduction in any format/media without specific permission when the material is reproduced accurately, is not used derogatorily or misleadingly, and the source is prominently acknowledged; explicitly identified third-party material is excluded. The same portal hosts Sikkim Herald, and the IPR Department identifies it as its official government paper. A live metadata-only enumeration of the newer IPR portal found 19 current Lepcha, 22 Mukhia/Sunuwar, and 5 Rai discovery candidates, while the State Portal exposes dated editions for all three. A bounded live State Portal catalog retained 29 unique canonical PDF URLs: 8 Lepcha, 11 Mukhia/Sunuwar, and 10 Kirat Rai. It rejected one duplicate and one non-PDF path. The searches advertise 193/167/174 records, but PageNo=1/2 returned identical URL lists, so the artifact is explicitly incomplete. It fetched no full PDF body and made only three eight-byte range probes against samples already full-hash-pinned by the prior policy. Every edition remains third-party-screening pending, train/eval blocked, and claim-grade false. Because the permission page is domain-specific, only sikkim.gov.in-hosted PDFs are catalog candidates; IPR-subdomain files remain discovery-only unless mirrored or separately permitted. This resolves the State Portal source-permission gate under a custom permission, not a public-domain claim. It does not restore the lost packs/models or cure the Kirat Rai pack's 100% text leakage. Next steps are full-hash acquisition and third-party screening of selected editions, followed by fresh edition-disjoint converter and native label audits. Evidence: outputs/sikkim-herald-source-rights-2026-07-15/, outputs/sikkim-herald-state-portal-catalog-2026-07-15/, docs/DATA_LICENSING.md, and docs/ATTRIBUTIONS.md.
  • Six Sikkim Herald editions were acquired, page-screened, and font-audited without creating labels. The source contract selected two State Portal-hosted editions per script: Kirat Rai 2024-09-26/2025-01-13 (109 days), Lepcha 2024-07-26/2024-10-04 (70 days), and Sunuwar/Jenticha 2024-04-12/2025-01-26 (289 days). All six full PDF hashes and all 26 page renders were frozen after an omac run. Reviewer 1 inspected every page at original render resolution: 25 pages have no explicit third-party marker observed, while both complete press-photo rectangles on Lepcha 2024-10-04 page 1 are bounded for exclusion because incidental storefront signage is visible. Both Lepcha PDFs have a structurally and visually blank page 5. This is a one-reviewer observation under the State Portal's custom permission, not a public-domain or government-authorship finding; reviewer 2 and adjudication remain null.
  • The revised live-layout converter audit reaches 100% aggregate coverage on both Kirat Rai and both Sunuwar issues. Kirat raw body-unit totals are 13,101 and 7,920. Exact comparison to the kirat rai font new program at xref 536 of official Unicode proposal L2/22-043R resolves the newly observed Herald T through canonical Z to U+16D6C; the isolated extracted C has an empty outline and normalizes to space, and original-resolution review establishes that the custom glyph under : is a double quotation mark. The pair has 91 observed semantic source-code matches, zero mismatches, and zero unresolved units. Sunuwar raw totals are 9,697 and 14,676. Both font subsets match the exact kirat1 program in the cited sunuwar_01348c63.pdf derivation source on every observed script code (76 summed matches, zero mismatches); backslash is an empty glyph, and the one U+201D extraction value visibly prints as a colon. The official proposal PDF hash is 88b1d4dfedd8b36a687533f3fa1ef0a97c42965fd595b28678ca8d7d17cb8f17 and its embedded reference font hash is bab945f8a5fe51f9401dc2a55a54ac99f3ba2b5071ac72324a05e7ff82cb4149; the Sunuwar reference PDF/font hashes are b3c001ff428de3b4e539c88d6d93566ae969a46b9ba947fe680fc1b68eb10456 and 484550543232f587fab3a5b3e900c922845c5ed44ef7bdd493cace4a5b6bce78, tied to MIT-licensed nepal-ttf2utf 0.2.0 at a158a8cbdbe230df6849afd9c9736bf31cb9c24b. The two 2024 Lepcha bodies are raster-composed, with 640 and 2,212 image placements, so converter coverage is N/A rather than 0%.
  • The joined bundle passes adversarial metadata audit but remains NOT_ADMITTED. Mutation tests reject any label-write claim, injected text/label field, unreconciled counter, font mismatch, or invented second reviewer. The frozen audit joins the exact source review, 26-page reviewer-1 record, revised converter artifact, and reference config; it records four live editions at 100%, two raster N/A editions, zero labels, zero native reviewers, no GPU/cloud/training/scoring, and no stored source or converted text. Exact and substring leakage are explicitly N/A with zero rows, not passed. Before any line can exist, the route still needs independent provenance reviewer 2/adjudication and native-script pixel/label review; only then may it create an edition-isolated train/eval split, run exact plus substring leakage audit, train, and compute codepoint CER. Artifacts: outputs/sikkim-herald-six-edition-source-review-2026-07-15-r11/, outputs/sikkim-herald-six-edition-manual-review-r1-2026-07-15-r11/, outputs/sikkim-herald-six-edition-converter-layout-audit-2026-07-15-r11b/, and outputs/sikkim-herald-r11b-adversarial-bundle-audit-2026-07-15-r11/.
  • A blinded Sikkim Herald provenance-review handoff is frozen, but no second review is claimed. The reviewer-2 packet links all 26 exact full-page renders at original resolution without copying images or exposing reviewer-1 decisions. Its byte-bound worksheet requires a distinct reviewer 2 who did not inspect reviewer-1 decisions before completion, a third distinct adjudicator, exact six-edition/26-page coverage, and one resolution for every disagreement and none for an agreement. Mutation tests reject prefilled decisions, reused identities, missing/duplicate pages, out-of-page exclusion geometry, label/prediction payload keys, and incomplete adjudication. The independently frozen packet audit verifies all 26 HTML links and hashes and reports PASS_READY_NOT_ADMITTED; the real worksheet remains blank, so reviewer 2, adjudication, native line review, rows, leakage, a model, and CER remain absent. Artifacts: outputs/sikkim-herald-six-edition-provenance-review-r2-packet-2026-07-15-r11/ and outputs/sikkim-herald-six-edition-provenance-review-r2-packet-audit-2026-07-15-r11/.
  • The recovered Newar-Devanagari whitespace rewrite is mechanically exact, but the underlying pack fails admission. A 64 GiB omac run independently reproduced all 1,820/565 train, 634/353 dev, and 427/142 test removals for the two narrowly defined artifact classes, matching all 6,763 normalized labels and leaving zero targeted patterns. All referenced PNGs decode and contain at least 16 dark pixels; there are 6,757 unique image hashes, four intra-train duplicate groups (six rows beyond the first), zero conflicting labels within those groups, and zero cross-split image hashes. The honest split audit fails: train versus dev+test has 18 exact rows and 575 rows with a shared 16+-codepoint substring (including exact), while dev versus test has zero exact but 136 substring rows. The recovered labels also lack source URL, PDF hash, page, bbox, date, and edition fields, and Gorkhapatra rights remain RISK. A deterministic non-native geometry review saw ink in all seven risk-selected samples but obvious neighboring-line fragments in two of the three widest crops; it makes no population-rate or label-faithfulness claim. No filtering, training, prediction, score, or CER was run in this audit round. Artifacts: outputs/newar-devanagari-normalized-claim-boundary-audit-2026-07-15-r12/ and outputs/newar-devanagari-normalized-visual-sample-review-2026-07-15-r12/.

Gurung Khema official-print reopening (2026-07-15 r13)

  • The former “synthetic-by-nature” source conclusion is refuted. Two official State Portal Sikkim Herald Gurung Editions contain running Gurung Khema print across eight pages and are 52 days apart. The 2022-03-01 PDF is SHA-256 e667bbd2eb4f6769209013fe569cdc37d3e1aff40a9f0121301b43a03afe5d14; the 2022-04-22 PDF is 2db16c7b7abe678e02103037397c7a35e21fb2fd34d4e4413d1979d1cf7f7c93. The pages contain 21,106 text-layer characters. April page 4 prints issue 4 while pages 1–3 print issue 5; the frozen source record preserves this anomaly. Reviewer 1 inspected every original-resolution page and observed no explicit third-party marker, but that is not a government-authorship or license-cleanliness finding.
  • The legacy font has a fail-closed Unicode 16 conversion path. Unicode proposal L2/22-096 states that the republished 2022 Herald uses the same legacy KHEMA font as the proposal and supplies the legacy key semantics. Both Herald PDFs use U+F0xx PUA re-encodings of those legacy bytes. Their 56 shared cmap codes have exact decomposed outlines and advances. For each edition, all 53 compared observed mapped glyphs rank first against the expected proposal glyph; the minimum observed binary-mask IoU is 0.490751. The converter handles direct keys, the Herald C/N/X printed-shape aliases, O/V combining-mark unifications, adjacent pre-base T/E reordering into Unicode logical order, and paired DD as double danda. It rejects ambiguous input rather than guessing.
  • Mechanical eligibility is 749 of 759 candidate lines, not a dataset. The March edition has 382 eligible and 6 rejected lines; April has 367 eligible and 4 rejected. Across both editions, the 10 rejections are four lines with no Gurung Khema character, five non-adjacent pre-base-medial sequences, and one isolated legacy danda unit. The audit serializes no source or converted strings and writes no crops or labels.
  • The route remains NOT_ADMITTED. The separately frozen reviewer-2 packet links all eight hash-bound source renders, exposes no reviewer-1 decisions, and contains a byte-identical blank worksheet. Its adversarial audit reports PASS_READY_NOT_ADMITTED. An independent reviewer 2 and distinct adjudicator, then native pixel-label review, crop construction, an edition/manuscript-isolated split, exact plus substring leakage audit, and codepoint-CER scoring are still required. There are currently zero labels, predictions, scores, training rows, or CER values. The joined r13 adversarial audit also pins every source/font/review/packet hash, rejects mutated counts, glyph ranks, blinding, payloads, and admission, and exercises the real freezer to prove that the blank worksheet fails before writing an output directory. Artifacts: outputs/sikkim-herald-gurung-khema-two-edition-source-review-2026-07-15-r13/, outputs/sikkim-herald-gurung-khema-font-semantic-audit-2026-07-15-r13/, outputs/sikkim-herald-gurung-khema-manual-review-r1-2026-07-15-r13/, outputs/sikkim-herald-gurung-khema-provenance-review-r2-packet-2026-07-15-r13/, and outputs/sikkim-herald-gurung-khema-provenance-review-r2-packet-audit-2026-07-15-r13/, plus the joined verdict at outputs/sikkim-herald-gurung-khema-r13-adversarial-bundle-audit-2026-07-15-r13/.

Corrected claim numbers (2026-07-16, fixes executed on omac)

  • Ol Chiki eval pack v3 (blank-guard builder, mechanism: white-fill invisible text in the source PDF): 772 clean lines (123 blank/near-blank dropped — the <16-dark-px guard caught 56 more than the census's fully blank 67). olchiki-v2 scores: public headline (CC0 issue) cp CER 0.0961 / exact 0.193 (n=358); September subset 0.0704 (n=414); combined 0.0826 / 0.2034 (n=772, internal until September license confirmed). Supersedes all prior Ol Chiki numbers.
  • Newar-Devanagari whitespace-normalized rescore (newar-ft-v1, frozen 1,139-line test, scored on omac): old artifact-bearing 0.0466; raw predictions vs normalized labels 0.0501 (the model reproduces training artifacts); both-sides-normalized cp CER 0.0441 / exact 0.2888 — the claim-comparable number (labels at outputs/newar-pack-v1-normalized-2026-07-15, utility src/ocrtech/devanagari_whitespace.py).

Tirhuta real substrate delivered (2026-07-17)

First-ever real Tirhuta training text: outputs/videha-tirhuta-silver-2026-07-17/ — 4,148 clean converted lines (labels.txt) from Videha issues (CC BY-SA 4.0, per-issue provenance), zero-unmapped-only policy. Extension-byte drop is the dominant loss (e.g. first issue: 1,235 extension-dropped of 7,622 examined) — building the conjunct extension-glyph map would multiply yield; per-issue numbers in the pack summary.json. Also staged and VERIFIED: Ol Chiki v3 fine-tune payload (outputs/olchiki-v3-ft-payload-2026-07-17/, 5,330 entries) — train is 242 real Public-Domain-Mark-issue lines + 5,000 synth replay, September 414 eval-only with a zero-intersection leakage audit on filename/line-id/label; READY-PENDING-APPROVAL in the GPU dossier (see outputs/overnight-consolidation-2026-07-16/gpu-batch-proposal.md, candidate 3 update). Operational note: the Codex quota briefly hit its limit on 2026-07-16 (reset message named 2026-07-22) but lifted the same night — fan-out resumed with the Tirhuta extension-map and Lepcha zero-yield lanes.

Lepcha zero-yield root cause (2026-07-17)

The Sikkim Herald Lepcha lane's 0 lines (vs Sunuwar 9,743 / Kirat Rai 3,699) is a data reality, not a bug: in all 18 editions the Lepcha article body is rasterized as image XObjects — there is no live Lepcha text layer (live text is English masthead/punctuation in Times New Roman / anonymized CID fonts; zero U+1C00-1C4F codepoints in sampled pages). The converter was never invoked. Full evidence: outputs/lepcha-zero-yield-diagnosis-2026-07-17/REPORT.md. Consequence: Lepcha real-line substrate requires a raster OCR/transcription lane (or stays synth-first per the headline plan); the builder should record raster_body_requires_ocr instead of wrong_font for such editions.

Tirhuta extension map + v2 pack: yield 2.6x (2026-07-17)

Codex built the Videha Tirhuta extension-glyph map (149 new mappings, each grounded in font glyph semantics + Pandey L2/11-175R; 2,131 converter tests pass) — branch agent/tirhuta-extension-map in nepal-ttf2utf, commit a5ba521. Rebuilt the pack on omac with the new converter: outputs/videha-tirhuta-silver-v2-2026-07-17/ — 10,696 clean lines (vs 4,148 in v1; 45,157 examined; extension-byte drops now 1, down from thousands). Pack verified (archive files == labels == manifest == 10,696), backed up to HF ampixa/ocr-tech-artifacts. Builder bug found during finalize: str.splitlines() splits JSON-legal raw U+0085 in provenance strings — fixed to split("\n") (patched builder stored in the pack dir). Remaining converter work: merge the branch to main after review.

Three FT payloads staged (2026-07-17)

Codex assembled per-script fine-tune payloads on the Ol Chiki v3 discipline (issue/page-boundary dev splits, four-way zero-leakage audits, sha-pinned manifests, training_launched=false):

  • Tirhuta READY: outputs/tirhuta-v2-ft-payload-2026-07-17/ — 9,765 real train / 844 dev (whole issue videha_01_02_08 held out), 10,639 entries, sha 55c18998...13e4. CC BY-SA 4.0 with attribution recorded.
  • Sunuwar BLOCKED-PENDING-LEGAL: outputs/sunuwar-v1-ft-payload-2026-07-17/ (8,422/514) — Sikkim Herald State-Portal permission + India §17(d) question unresolved; assembled but not trainable yet.
  • Kirat Rai BLOCKED-PENDING-LEGAL: outputs/kiratrai-v1-ft-payload-2026-07-17/ (2,865/193) — same gate.

Verified: no synth replay pools or warm-start checkpoints exist for these three scripts anywhere in outputs/ or HF artifacts (the native-script wave built substrate, not models; only Ol Chiki trained). Consequence: the proven synth-pretrain-then-real-FT recipe needs a Tirhuta synth pack before the Tirhuta payload trains warm; alternatively from-scratch real-only must be an explicit dossier decision.

Tirhuta synth-v1 pretrain pack rendered (2026-07-19)

The missing pretrain stage for Tirhuta is now built: outputs/tirhuta-synth-v1-prep-2026-07-17/tirhuta-synth-v1-rendered.tgz (sha 2bf91b49...1327, 48,960 PNGs = 8,160 texts x 6 degradation profiles, 7,997 train / 163 eval texts, zero normalized collisions with the real FT dev set, Noto Sans Tirhuta OFL only, RAQM shaping verified, dark-pixel guard min 102). Rendered on omac; archive re-extracted and hash-verified; backed up to HF. Engineering notes: macOS Pillow lacks RAQM — fixed via Xcode CLT + brew libraqm + DYLD_FALLBACK_LIBRARY_PATH exported INSIDE the script (SIP strips DYLD_* across /bin/zsh exec); audit had a /tmp vs /private/tmp resolve() mismatch, patched in the pack's audit script. Tirhuta training chain is now fully staged: synth-v1 pretrain -> tirhuta-v2-ft real payload (dossier candidate 5); GPU run still human-gated.

Next Work

  1. Post-recovery next (2026-07-05): the spike verdict keeps the Paddle det + Rows-LTR + y/conf70 chain as the backbone. Phase-2 pivot, measured 2026-07-05: decomposing the 0.4134 filtered-baseline page CER on the 16-page pack showed the standing "region selection + furniture" assumption is wrong — deleting every unmatched prediction (perfect furniture/region filtering) changes CER by ~0 (0.4152), while reordering the matched lines into reference order alone drops it to 0.0378 (10.9×). Reading order is 91% of page CER. Geometry heuristics cannot recover it (Columns LTR 0.799, PP-Structure-region banding 0.702, PP-Structure native order 0.797) because the reference order is the PDF's article-box emission order, not a row/column pattern. Three follow-up experiments (all 2026-07-05, artifacts under outputs/reading-order-model-v1-2026-07-05/ and outputs/line-merge-v1-2026-07-05/) then closed the question:

    • Learned reading-order model v1: refuted. LightGBM pairwise comparator on the silver span_bbox order supervision regressed to 0.799 — the Devanagari packs preserve PDF article-box emission order (not row-major) while the Limbu eval reference is a strict y-sort; wrong supervision for the target. Leakage-clean; kept as a negative result.
    • Detector-fragmentation hypothesis: refuted. Det boxes ≈ 1:1 with reference lines (mean best-IoU 0.68; the widest lines average 1.05 fragments). Tuned merge (gap 0.6×h) merges ~3 boxes/page, CER flat. Naive merge (gap 1.0×h) bridges the 40 px column gutters and collapses det F1 to 0.71 — gutters are narrower than line height on these pages.
    • Root cause found: the reference linearization is our own artifact. build_limbu_gorkhapatra_all_corpus.py:209 sorts spans strict y-then-x across a 6-column layout; det-vs-PDF y-jitter of a few px flips column interleaving constantly (pair accuracy 0.998, page CER still 0.30–0.41). Pure-y prediction sort gives a free win (0.4134 → 0.3006), and one shared deterministic column-major linearization applied to both reference and prediction yields page cp CER 0.0451 — at the content floor (~0.054: recognition 0.026 + 211 missed / 87 extra lines). Page reconstruction on this pack is solved up to linearization definition; the honest next metric is page-eval v2 with column-major linearization on both sides (a documented metric change — not comparable to the old 0.41-scale numbers), and the production assembly should emit column-major order. Remaining real levers after v2: det recall/precision (211/87), recognizer (0.026), and native review for claim-grade.

    Page-eval v2 + Column Major assembly shipped 2026-07-05: the shared linearization lives in src/ocrtech/linearization.py (tests in tests/test_linearization.py); tools/apply_page_reading_order_strategy.py gained mode "Column Major"; tools/run_page_ocr_eval.py gained --page-linearization {native,column-major} (recorded in page-eval-run.json, per-sample fallback to native recorded when a reference lacks line geometry). Validated on the 16-page pack: Column-Major assembly + v2 scoring gives page cp CER 0.043 / grapheme 0.067 (outputs/limbu-column-major-2026-07-05-score-v2/); the native path is regression-identical (0.413/0.437). v1-scale and v2-scale numbers must never be compared directly.

    Newar page-eval v2 (second-language proof, 2026-07-05/06): a 19-page Newar page pack built from the rebuilt harvest (tools/build_newar_gorkhapatra_silver_page_eval.py, outputs/newar-page-eval-v2-2026-07-05/pack/) and scored through the identical non-oracle chain (Paddle det → recognizer → Column Major → y/conf70 → page-eval): newar-ft-v1 native 0.778 → v2 0.199 page cp CER (grapheme 0.279; det F1 0.912; matched-line cp CER 0.0905) — the column-major linearization story replicates exactly on the second language; the Newar residual is recognition + det-crop domain gap, not reconstruction. Two follow-ups surfaced and both were fixed 2026-07-06: (a) the deployed limbu-devanagari-ppocrv5 export was unusable as a general Devanagari recognizer (409-byte dict missing anusvara, ii-matra, ṇa…; emitted garbage — its page numbers were a formality, not a model comparison); the recovered deva-v2 training checkpoint was re-exported with its correct 144-char devanagari-v2.txt and verified to reproduce the recorded pre-crash Newar baseline exactly (cp CER 0.062229 / grapheme 0.13120 / exact 0.158 on the recovered 1,139-line frozen test); the Space now serves it as models/deva-v2 (export also at outputs/deva-v2-reexport-2026-07-06/ and on the HF dataset). (b) The Newar pack builder's digest mismatch was a computation-path divergence from the eval harness's recomputation — builder fixed (9 tests green), rebuilt pack at outputs/newar-page-eval-v2-2026-07-06-digestfix/ scores with zero digest warnings and identical CER (0.19918).

    Phase 3 done 2026-07-05: config-driven tools/run_book_ocr.py landed (det → script-route → per-script recognizers from configs/nepal-language-pack-pipeline-v1.json → Column Major default → filter → Unicode/MD export + review queue, resumable, preflight validation; smoke on 2 rebuilt Limbu pages: 606 lines kept / 25 queued). The rebuilt Limbu pack is 2× the lost one (16 pages / 4,872 labels).

    ft-v2 retrain result (2026-07-06): new champion candidate. 40 epochs on the L4 (~8.5 GPU-h), 46,753 rows (25,065 synth + 2,711 real ×8 replay), PP-OCRv5 mobile generic base, selection on real dev (best epoch 39). On the frozen 902-line test (dates 2082-03-24 / 2082-08-12 / 2082-12-25, by-date disjoint, leakage-gated), scored through the identical local CPU path as the deployed champion: ft-v2 cp CER 0.0047 / grapheme 0.0098 / exact-line 0.859 vs champion 0.0138 / 0.0284 / 0.637 — ~3× lower CER, +22 pts exact-line. The gap is if anything understated: the old champion's training pack drew on the old harvest and may have seen these dates; ft-v2's test isolation is proven. We also now own a warm-startable training checkpoint again (outputs/limbu-ft-v2-run-2026-07-06/checkpoints/best_accuracy.*; export at .../export/). Silver labels, diagnostic, not claim-grade. Acceptance gate PASSED (2026-07-06): ft-v2 through the full non-oracle 16-page chain (same detector, Column Major, y/conf70 filter, page-eval v2): page cp CER 0.0272 / grapheme 0.0347 vs champion 0.043 / 0.067; matched-line cp CER 0.0090 vs 0.0266 (−66%); det F1 unchanged (0.9650 vs 0.9657, same detector). 3,725 of 4,699 matched lines exact. Gate artifacts: outputs/limbu-ftv2-page-gate-2026-07-06/. Real Gorkhapatra newspaper pages now read at ~2.7% codepoint error in human-readable column-major order, silver-diagnostic boundary unchanged. Remaining step: package ft-v2 (model card, hashes) and swap the Space bundle — a production-facing change, pending user confirmation.

  2. Finish the 2026-07-04 full harvest and inventory Done 2026-07-04: 394 language-page hits, 50 languages, inventory manifests at outputs/language-pack-inventory-full-2026-07-04/.

  3. Per-language silver packs via the existing nayanepal converter Done 2026-07-04 (tools/build_language_pack_silver_line_packs.py, tests green): the batch-1 run produced 116,287 kept silver lines across 50 languages, zero errors, with per-line provenance (PDF hash, page, bbox, raw legacy bytes, converted text, image hash) at outputs/language-pack-silver-lines-2026-07-04/; spot-checked conversions read as clean, language-plausible Devanagari. The batch-2 run over the older 589 page hits kept 183,324 more lines (49 languages, zero errors) at outputs/language-pack-silver-lines-batch2-2026-07-04/ — 299,611 per-language-attributed silver lines total across the two batches (~2.4 GB of crops). Run the tool with uv run --python 3.11 --with pymupdf --with npttf2utf — nepal_ttf2utf needs the npttf2utf base-map package at runtime, and without it every span fails (cleanly, with per-span error records, but kept=0). Honest caveat: lines are page-attributed, not language-verified — a language page also carries Nepali furniture/boxes, and pages whose body is not nayanepal Devanagari (Urdu, the Tibetan-script pages) yield mostly furniture lines. A language-ID/lexicon filter before per-language recognizer training is the follow-up gate.

  4. Newar track (second focus language): add the Newar registry entry (Devanagari + Prachalit rows); build the Devanagari-written Newar recognizer eval/train split from the 6,800 harvested silver lines (by-date split, frozen test); keep native Prachalit as the separate longer arc — real printed-register Nepal Lipi harvesting, not more transliteration, is the lever.

  5. Classify Tibetan-script pages (Hyolmo/Mugal/Bhote) and decide converter vs native Unicode.

  6. Generalize the pipeline wiring Done 2026-07-04: all three Limbu-locked spots generalized (shared script table, filter keys, --required-script preflight), tests green; configs/nepal-language-pack-pipeline-v1.json holds the per-language wiring.

  7. Target-region/furniture filtering stage for Paddle page/book OCR, scored against the current baseline on the 8-page Limbu pack and at least one Devanagari-language pack (Newar is the natural candidate given the focus order).

  8. Only after page reconstruction improves, export high-quality real/silver line crops back into recognizer training, per language.

  9. Month-batch the archive back-harvest toward the full ~1,750-article depth, storing the mirror on the remote 2 TB volume.

Mistakes To Avoid

Do not use Naya Nepal blog images as training/eval evidence when the PDF exists. Do not use wide epaper date windows — the endpoint silently caps at 30 PDFs per query; weekly windows only. Do not treat unnormalized Gorkhapatra labels as distinct languages (थारु vs थारू). Do not treat the nayanepal text layer as Unicode — it is legacy Preeti-family and needs the converter before any silver label is real. Do not report Paddle norm_edit_dis as CER; real CER comes from tools/eval_metrics.py micro CER over canonical predictions. Do not treat synthetic data as claim evidence. Do not call Gurung Khema synthetic-by-nature now that official Herald print is frozen, and do not promote its 749 mechanically eligible candidates to rows before independent provenance, native label, crop, split, and leakage review. Do not promote the targeted+Unicode checkpoint or v3 dictionary warm-starts. Do not spend long blocks on generic eval-harness work unless it directly improves the book/page OCR loop. Do not interpret a syntactically valid approval/auditor identity as cryptographic proof; retain independent custody or signed runner evidence for a production claim. Do not extend the State Portal's Sikkim Herald permission to the separate IPR PDF origin; use ipr.sikkim.gov.in as discovery-only unless a file is mirrored or separately permitted. Do not frame video-to-page work as OCR scope; reduce video to stills first. Do not present page OCR as solved: matched-line recognition is strong, but full-page CER is still dominated by reconstruction and region selection.