Download ocr-tech/RESEARCH.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 101 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/RESEARCH.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/RESEARCH.md
-
curl -L -o RESEARCH.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/RESEARCH.md
Research State: Nepal Language Pack OCR
Updated: 2026-09-24
Devanagari recognizer fine-tune on k2 (2026-09-24)
Goal (owner steer): make the Devanagari route of the live demo (ampixa.thapa-ashish.com.np,
k2 OCR worker) read citizenship-certificate-style and general documents properly. Resource
inventory: docs/DEVANAGARI_RESOURCES_INVENTORY.md.
Base: official
devanagari_PP-OCRv5_mobile_rec(568-entry dictionary incl. all Latin letters and digits); replaced ourdeva-v2on the demo the same morning.Training path: Paddle CPU training on k2 (Apple M2 Ultra) runs at
2 img/s, so the same network is trained with the70-90 img/s), exported to ONNX, and served through PaddleXtransformersPyTorch port on MPS (create_model(engine="onnxruntime"). Code:tools/deva_rec_torch/. Preprocessing is bit-matched to PaddleX (98.8% line parity).Data: real Gorkhapatra print re-cut into PDF-aligned chunks (
tools/build_gorkhapatra_devanagari_chunks.py, edition-disjoint: test 2082-03-24, 2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11); synthetic citizenship/office/real-text lines in 44 OFL font families (tools/render_devanagari_form_synth.py); NVIDIA OCR-Synthetic-Multilingual-v1 English (CC BY 4.0) plus Liberation/DejaVu/Lato English renders.Converter bug found and fixed: npttf2utf 0.3.7 mis-decodes the nayanepal alternate क्र ligature
j|m(-> व्र, and प -> फ / straym). 3,426 of 114,452 spans (3%) were affected (कार्यक्रम, प्रक्रिया, शुक्रबार, ...). Fixed in the chunk builder (RAW_FIXES); labels of all earlier Gorkhapatra-derived packs (incl.data/gorkhapatra-devanagari-real, the Newar and language-pack silver packs built from the same converter) carry the bug. Upstream fix still needed in nepal-ttf2utf.Results, corrected labels, served path (PaddleX), codepoint CER / exact line (
~/ocr-deva-ft/preds/final_comparison_v3.jsonon k2):Set official v2 v3 (deployed 13:07) E1v4 real chunks, held-out editions (n=3000) 5.13% / 0.47 0.69% / 0.91 0.57% / 0.93 E7 real chunks containing क्र (n=530) 5.52% / 0.33 4.28% / 0.18 0.86% / 0.89 E1b real full lines, independent labels (n=1500) 7.69% / 0.13 2.62% / 0.53 2.57% / 0.54 E2 Newar test (n=1139) 9.11% / 0.06 3.53% / 0.35 3.54% / 0.35 E3 synthetic citizenship-ne, unseen fonts (n=800) 6.78% / 0.56 0.84% / 0.93 0.75% / 0.94 E3 synthetic office-ne, unseen fonts (n=300) 7.77% / 0.54 0.62% / 0.94 0.92% / 0.92 E3 synthetic citizenship-en (n=300) 1.94% / 0.83 0.53% / 0.96 0.64% / 0.95 E4 English, unseen Noto Sans (n=120) 2.57% / 0.62 1.58% / 0.73 1.64% / 0.72 E4 mixed Nepali/English (n=120) 6.78% / 0.36 2.88% / 0.68 2.67% / 0.70 E6 NVIDIA English, held-out pages (n=419) 10.6% / 0.71 2.99% / 0.84 2.84% / 0.84 E5 handwriting probe (n=450) 64.7% 60.5% 61.0% (unusable) v2 learned the converter's क्र->व्र error (402 of 530 E7 predictions contained व्र); v3, a 6,198-step continuation on the corrected
chunks-v4, removes it (2 of 530, same as the official model). Deployed as~/ampixa-demo/models/ocr/space/models/devanagari-ft-v3; v2 and the official model remain on disk for rollback. E3 is synthetic from the same generator as training (different seed, held-out fonts) and overstates real-certificate accuracy; no real citizenship certificate has been tested. Real labels are PDF-text silver, not human-reviewed. Page OCR at 144 dpi garbles small text; at 288 dpi it reads cleanly.Next: real citizenship samples (with consent) for a true eval; handwriting data (none licensed yet); upstream the
j|mfix into nepal-ttf2utf and rebuild the other Gorkhapatra-derived packs; the demo's "Rows LTR" reading order interleaves newspaper columns.
Page pipeline on real government letters (2026-09-24, smoke test)
Two public notices from ocr.gov.np (a scanned Lalitpur District Court letter tilted 2°, and a
Department of Industry letter), rendered at 200 dpi; 43 printed lines transcribed by hand
(/ocr-deva-ft/layout-probe/tools/deva_rec_torch/page_probe/: GT, probe scripts, results; PDFs on k2 `; not a frozen pack, not claim-grade). Same detector config and v3 recognizer as the k2 worker; micro CER via tools/eval_metrics.py`:
| Crop of each detected line | cp CER | grapheme CER | exact |
|---|---|---|---|
| worker today: axis-aligned bbox of the quad + 12 px | 21.4% | 31.7% | 0.40 |
rotated min-area quad (PaddleX CropByPolys), no pad |
3.6% | 5.5% | 0.49 |
| rotated quad + 6 px | 2.9% | 4.2% | 0.65 |
| rotated quad + 12 px | 8.1% | 13.3% | 0.42 |
- The worker's crop, not the recognizer, is the dominant error on real letters.
_box_from_polygonflattens the detector quad to an upright box; on a 2° tilt a full-width line spans ~40 px vertically, so the crop swallows neighbouring lines (court-letter body lines read at confidence 0.24-0.57). Upstream PaddleOCR uses rotated crops; our port dropped them. Line-level evals (E1-E7) cannot see this. Not yet deployed: needs a page-level eval. - PP-DocLayoutV3 on the same pages: QR codes found (image, 0.95/0.96); emblems found
(header_image); 0 of 2 stamps labelled
seal(court seal -> header_image 0.65; the Industry stamp printed over the title not detected at all); signatures merged into the printed-nametextblock (0.52); no class for handwritten fill-ins; several printed lines (subject, dates, reference numbers) not boxed at the default threshold. Use layout to mask non-text regions, never to decide what gets OCR'd. - Signature/stamp crops produce junk text at recognizer confidence 0.35-0.71; damaged but real lines (tick mark, strike-through) score 0.74-0.82, so confidence can flag but not drop.
train.pynow supports--guard-dev(print/synthetic/English dev sets may not regress more than--guard-tolover step 0); the plain dev mean let a 5-point handwriting gain hide a 1-point print loss.run_train_v4.shon k2 uses it.
Paper-rewrite verification pass (2026-07-19): corrections to this document
While rewriting paper/DRAFT.md to cover the language pack, every number was
re-verified against on-disk artifacts. Four values recorded earlier in this
file do NOT match any surviving local artifact and must not be cited:
- Newar "both-sides-normalized cp CER 0.0441" — no local artifact; the
omac rescore output was never mirrored. The surviving verified score is
0.04655 cp / 0.09569 grapheme / 0.2450 exact (artifact-bearing labels,
outputs/newar-ft-v1-verify-omac-2026-07-07/metrics.json, n=1,139). - Ol Chiki "public headline cp CER 0.0961 / n=358" — no local score
artifact exists for the 358-row public-issue subset. Locally verified:
v2 on the 772-line v3 pack scores 0.08264 cp / 0.2034 exact
(
outputs/olchiki-v2-run-2026-07-07/score-real-print-v3pack/metrics.json); the documented public-issue historical diagnostic remains 0.1018 (n=366) perdocs/DATA_LICENSING.md. The v3 358-row candidate was never scored. - "299,611 per-language-attributed silver lines" — refers to the lost
pre-crash batches. The rebuilt batches on disk total 306,971
(122,882 + 184,089,
outputs/silver-lines-rebuild-b{1,2}-2026-07-05.log). - Ol Chiki v3 FT payload "5,330 entries" — the payload's own assembly manifest says 5,307 total images (242 real train + 5,000 synth + 65 real dev).
Also noted: the 8-language Devanagari scorecard (grapheme CER 0.075–0.129, 2026-06-18) lost its backing artifacts in the crash; it is text-only history now and the paper deliberately does not report it.
Machine-death recovery (read this first, 2026-07-05)
The Mac died 2026-07-04; the repo now lives at
/Volumes/TRANSCEND/recovery-backup-20260704-172213/github/llm/ocr-tech
(symlinked from the old /Users/cdjk/github/llm/ocr-tech path). outputs/
and runs/ are lost; code, .git, docs, spaces/ model exports
(sha256-verified, champion included), and most of data/ survived. Every
outputs/…/runs/… path below therefore refers to a lost artifact unless
docs/RECOVERY_LEDGER.md maps it to a rebuilt one — read that ledger for the
full survived/partial/lost inventory, recovery routes, and the rebuild policy
(rebuilds go to new dated dirs). Batch-1 re-harvest launched 2026-07-05.
Blocked on user credentials: AWS SSO (training checkpoints, newar-ft-v1
weights) and HF auth (Space cross-check).
Overnight lane results (2026-07-16/17)
These lanes delivered mechanically filtered silver substrate, not claim-grade training or evaluation data. The counts below establish file and archive parity; they do not replace native label review, edition-isolated leakage checks, or a real-domain score.
The Sikkim pack at outputs/sikkim-silver-packs-2026-07-16/ contains 9,743
Sunuwar/Jenticha clean real-page silver lines and 3,699 Kirat Rai clean
real-page silver lines. In each case the summary.json clean-line count,
on-disk labels.txt line count, and non-AppleDouble .tgz image-member count
are identical. Sampled Sunuwar labels contain Jenticha characters in
U+11BC0..U+11BFF, and sampled Kirat Rai labels contain Kirat Rai characters in
U+16D40..U+16D7F, without non-ASCII characters outside the expected target
block. Each summary records the eval manifests it checked. None of those eval
editions occurs in the discovered source list, so the recorded
excluded_editions: [] is consistent rather than evidence of an unperformed
check.
Lepcha takes a different route. The diagnosis at
outputs/lepcha-zero-yield-diagnosis-2026-07-16/REPORT.md shows that the new
edition bodies are raster content and that the live PDF text is ordinary
header/punctuation material, so there is no converter alias or byte-map fix
that can recover the body. The converter lane is therefore closed and Lepcha
moves to a synth-first recognizer route, retaining the 132-crop real pack
recorded in data/lepcha-real-eval-pack/README.json as a real evaluation
anchor. That pack is still machine-labelled and human verification remains
pending, so it is diagnostic rather than claim-grade.
The Gorkhapatra full-depth backfill now covers every month from March 2024
through July 2025. The figures come from each month's
gorkhapatra-language-pages.json; silver-kept comes from the corresponding
silver summary and was independently matched to the .tgz member count.
| Month | Items | Aligned | Hits | Silver-kept |
|---|---|---|---|---|
| 2024-03 | 1,776 | 62 | 120 | 27,668 |
| 2024-04 | 1,731 | 61 | 129 | 21,429 |
| 2024-05 | 1,666 | 62 | 128 | 24,612 |
| 2024-06 | 1,601 | 60 | 118 | 21,924 |
| 2024-07 | 1,538 | 61 | 120 | 28,301 |
| 2024-08 | 1,473 | 60 | 120 | 24,534 |
| 2024-09 | 1,408 | 60 | 122 | 32,804 |
| 2024-10 | 1,343 | 53 | 103 | 25,055 |
| 2024-11 | 1,278 | 52 | 106 | 30,788 |
| 2024-12 | 1,213 | 61 | 121 | 39,115 |
| 2025-01 | 1,148 | 62 | 124 | 40,526 |
| 2025-02 | 1,083 | 56 | 112 | 35,391 |
| 2025-03 | 1,018 | 59 | 117 | 35,482 |
| 2025-04 | 953 | 57 | 113 | 34,815 |
| 2025-05 | 888 | 59 | 118 | 36,878 |
| 2025-06 | 823 | 60 | 120 | 37,836 |
| 2025-07 | 798 | 32 | 62 | 18,428 |
| Total | 21,738 | 977 | 1,953 | 515,586 |
All monthly audit warning arrays are empty, all archive counts match their silver summaries, and no source PDF remains under any monthly output. This is a delivered backfill substrate, not a statement that its page-attributed silver labels are language-verified.
Detector remediation remains active. The scale-normalized tile build is
staged/running on omac; the local evidence at
outputs/det-remediation-2026-07-16/REPORT.md ends at the completed code,
config, and handoff. The GPU gate remains closed until the returned build
summary and tile manifest pass the recorded re-audit checklist.
Current Goal
The end goal is an OCR system covering the multiple languages of the Nepal pack, built out of Paddle, with Limbu and Newar (Nepal Bhasa) as the first two focus languages (user steer, 2026-07-04). The system should accept real book/page images or PDFs for any language in the pack and output editable Unicode text with page, line, crop, confidence, and provenance evidence preserved. Limbu stays the proven template — it has the strongest recognizer and the deepest harvest. Newar is the deliberate second: it has its own native script (Prachalit), real cultural demand, and is currently the weakest trained recognizer, so it exercises every part of the method that Limbu already proved. Every stage that was built Limbu-specific must generalize: harvest, silver-label extraction, recognizer training, script routing, page/book OCR, and eval. The product is the full pipeline: page detection, script-aware routing to per-script recognizers, target-region and furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and review artifacts for hard lines. YOLO line detection stays a diagnostic/probe; Paddle-first is the production story.
The Language Pack
Two inventories exist and they are not the same thing:
- The registry
languages/nepal-v1.jsontracks 18 target languages (17 Nepal + English) with scripts, fonts, dictionaries, and corpus status. - The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found
50 normalized languages (53 raw labels) in 400 articles over ~7 months (see the
harvest section). Labels include all registry languages plus Chamling Rai, Bahing,
Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo,
Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi,
Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more.
Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा,
ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) —
tools/build_gorkhapatra_language_pack_inventory.pyowns that normalization and the label→registry mapping.
Newari / Nepal Bhasa writing-system boundary (2026-07-15)
Newari support is explicitly four-profile, not a single generic "Newa" bucket:
| Profile | Source encoding | OCR transcription | Current capability |
|---|---|---|---|
| Newar / Prachalit | Unicode Newa, U+11400..U+1147F | Native Newa | Route, training candidate, diagnostic checkpoint, and synthetic runtime smoke registered; no claim-grade real-domain CER |
| Bhujimol / Bhujinmol | No separate Unicode script; configured as a Newa-codepoint style profile | Newa plus mandatory source-style metadata | Route, two acquired manuscript groups, and reviewer reference registered; zero qualified decisions, admitted labels, model, or CER |
| Ranjana | Unencoded in Unicode 17.0 | Newa transcription plus mandatory source image and Ranjana metadata | Image route, two acquired manuscript groups, and reference font registered; zero qualified decisions, admitted labels, model, or CER |
| Devanagari | Unicode Devanagari, U+0900..U+097F plus U+A8E0..U+A8FF | Native Devanagari | Route, diagnostic checkpoint/runtime smoke, and one permission-covered official Newa-edition acquisition candidate registered; source script and real-line quality remain unaudited |
Unicode's current Newa chapter and code chart define the encoded Newa route. WG2 document N4555 describes Bhujimol/Bhujinmol as graphically distinct and discusses possible plain-text unification with Newa; that does not justify visual-model reuse. Ranjana is still represented by a preliminary encoding proposal, so this project does not invent a Ranjana Unicode range.
The canonical runtime profiles live in src/ocrtech/newari_scripts.py. The
registry and book-OCR pipeline declare four separate recognizer references and
preserve the source encoding/output policy in run metadata. The frozen audit at
outputs/newari-four-writing-system-support-audit-2026-07-15-r1/ passes all four
structural bindings while reporting model_ready=0/4 and
four_script_ocr_claim_ready=false. "Supported" here means explicit input,
profile, routing, and output semantics; each profile still needs at least 200
license-clean, pixel-label-audited, leakage-controlled real lines and a frozen
real-domain evaluation before delivery or quality claims.
The r2 license-first discovery audit records three Bhujimol and three Ranjana
source candidates without admitting any pixels. The most promising Bhujimol
item is Asa Archives item 1147:
its Internet Archive metadata has an uploader-applied CC0 URL, 63 scanned
pages, and 325 automatically detected hOCR line boxes. Those boxes are
acquisition-scale metadata, not labels. Uploader authority, institutional
accession, exact-file SHA-256, qualified Bhujimol review, pixel faithfulness,
and transcription mapping all remain open. For Ranjana, the Smithsonian's
Karandavyuha Sutra record
specifically marks image FS-6314_06 CC0 and its IIIF metadata identifies
Ranjana, but the manifest contains
only one canvas; the 77 physical folios are not 77 downloadable pages. British
Library EAP790/21/121
identifies 12 Bhujimol TIFFs but restricts access to research use, and
EAP1296/2/217 exposes 13
Ranjana images under noncommercial terms. The Nepalbhasa Manuscripts in
UK archive states that copies and
transcriptions are openly accessible but supplies no explicit reuse grant.
The frozen audit is outputs/newari-source-candidate-audit-2026-07-15-r2/;
its pass means the catalogue is fail-closed, not that any source, model, or OCR
claim is ready.
The r3 provenance crosswalk matches item 1147 to the uploader's Asa Kuthi
catalog spreadsheet at worksheet aiStuidoOutput, row 1170: DPN 1147,
Bhāsvati, NRN ज्योति - २७१, RN 2919, Sanskrit/Nepalbhasa, N.S. 528, and
33.3 × 4.1 cm. The spreadsheet SHA-256 is
8484dbe95358b83d7d78d5d1752fafac5feeecfc007ccf5d1fe8b6ea3ce9bffd.
This disambiguates a separate row whose RN, rather than DPN, is 1147. It does
not attest Bhujimol because the matched catalog script field is blank, and it
does not prove that the Internet Archive uploader was authorized to apply
CC0. The exact authorization and reuse questions are staged in
docs/NEWARI_ASA_1147_RIGHTS_REQUEST.md but have not been sent. The r3 frozen
audit therefore remains PASS_CATALOG_INVENTORIED_ZERO_DATA_ADMITTED with no
review manifest ready.
The r4 alternative-source sweep found a materially stronger Bhujimol route in
Wellcome Collection. The official manuscript
record identifies MS Indic epsilon 1, and Wellcome's editorial account of the
object
explicitly says it uses Bhujimol. Its complete IIIF
manifest has
217 unique canvases/image services, SHA-256
596296750114948b39da2252b5a7b7c0738287dadcf7aed32939bfc29980c6d4, and
a Public Domain Mark statement that expressly permits commercial copying,
modification, and distribution. Wellcome describes 208 leaves with only six
illustrated. Five 2000-pixel-wide probes at canvases 1, 50, 100, 150, and 210
decoded successfully; canvas 1 was a cover/label view, while the other four
showed dense multi-block palm-leaf text. This is not a script-expert or label
audit. Because the current host is an 18GB Mac, the repository freezes only a
metadata acquisition manifest and an omac handoff; it does not bulk-download
the 217 canvases. The whole manuscript must remain on one side of any split,
so a separate manuscript is still required for leakage-proof training.
The r5 Ranjana follow-up resolved Wellcome MS Indic delta 33 without
inflating selected images into a complete manuscript. The official physical
record is closed-store only and
has no digital item; the predictable IIIF presentation
URL returns
404. Four exact digital-image records identify Ranjana and carry CC BY 4.0,
but a prediction-free 1200-pixel probe shows one two-leaf overview, one
partial-leaf environmental photograph, and two close-up miniature crops whose
content overlaps the overview. The records therefore provide zero independent
complete pages, not four. Their titles say AD 1653 while one description says
transcribed in 1635; both values are retained. The frozen catalogues at
outputs/newari-wellcome-ranjana-source-catalog-2026-07-15-r5/ and
outputs/newari-source-candidate-audit-2026-07-15-r5/ admit zero pixels,
labels, rows, models, or metrics. A different complete Ranjana manuscript is
still required at the r5 boundary.
The r6 BDRC sweep resolves that complete-source discovery gap without claiming
a dataset. BDRC physical instance MW1NPL132, Ārya nāmasaṁgīti, is marked
only ScriptRanj with print method Manuscript; linked scan W1NPL132 is
AccessOpen and StatusReleased, and image group I1NPL132 exposes 21 unique
IIIF canvases/services. The manifest SHA-256 is
a3c49a3fd68f30f0b4610a2f0c3d6432fd189caae5c371a0d482ae97fb02d11d.
BDRC's official access policy classifies scanned original manuscripts as
public domain and permits sharing/adaptation for any purpose. The manifest
itself has no license field; the work-record CC0 marker applies to metadata and
is explicitly not treated as a pixel license. Five bounded 1200-pixel probes
decoded: canvases 1 and 21 are cover/note material, while 5, 10, and 15 show
dense visually Ranjana-like running text. Those observations are not qualified
script decisions or a line-count audit. The frozen catalog at
outputs/newari-bdrc-ranjana-source-catalog-2026-07-15-r6/ records 21 exact
remote resources, zero admitted pixels, zero audited lines, zero labels, and a
prediction-free omac handoff. All pages remain one manuscript group; a
distinct manuscript is still required for a manuscript-disjoint train/eval
split.
The r7 remote-acquisition round executed that BDRC handoff on
omacs-MacBook-Pro (68,719,476,736 memory bytes), without OCR, training, or
scoring. All 21 full-resolution JPEGs were fetched, SHA-256 recorded, and
decoded at their exact manifest dimensions (all width 1200, heights
1676–1769); the capture totals 6,680,141 bytes and has 21 unique byte hashes.
The source pixels remain on omac; the orchestration Mac mirrors only the
metadata, hash list, dimension list, and rights/manifest evidence. A dynamic
WordPress detail was caught rather than waived: BDRC's policy HTML randomizes
CSS class-number suffixes across hosts, so raw page hashes differ while a
narrow normalization of only those suffixes yields the same SHA-256
99977ba3f0cb00da828a3833e6649a39c89c9f8e928a7127eeac1c35cd48d60e
and the required rights sentences remain present. The local adversarial audit
at outputs/newari-bdrc-ranjana-remote-acquisition-audit-2026-07-15-r7/
verifies the returned metadata bindings and creates a 21-row blank review
worksheet. It explicitly cannot recompute the remote image hashes without the
pixels. Qualified page decisions remain 0/21, audited lines remain zero, and
no label, admission, model, or metric exists.
The r9/r10 remote-acquisition round executed the independent Wellcome
Bhujimol handoff on the same approved host, again without OCR, training, or
scoring. All 217 requested review JPEGs were fetched, SHA-256 recorded, and
decoded; the capture totals 58,471,370 bytes and has 217 unique byte hashes.
Wellcome's full/2000,/... IIIF derivative decoded at width 2000 for 214
canvases and width 2001 for canvases 214, 216, and 217. The runner failed
closed on the first 2001-pixel result, then the replay narrowed the accepted
contract to exactly those observed widths and froze every exact dimension.
The source pixels remain on omac; the local mirror is metadata-only. The
adversarial audit at
outputs/newari-wellcome-bhujimol-remote-acquisition-audit-2026-07-15-r10/
verifies the returned bindings and creates a 217-row blank qualified-review
worksheet. Qualified page decisions remain 0/217, audited lines and labels
remain zero, and the entire manuscript is still one indivisible split group.
A second Bhujimol manuscript is required for manuscript-disjoint evidence.
The r12/r13 second-source round catalogs a distinct manuscript group for both
Bhujimol and Ranjana without admitting data. The exact Wikimedia Commons JPEG
of Cambridge MS Add.2137 is photographer-authored and CC BY-SA 4.0. It
contains four folio photographs; the first three selected regions are visually
crosswalked to Cambridge canvases 48–50 (28r, 28v, 29r). Cambridge's
official manifest independently calls all three hands
hooked Nepālākṣarā (Bhujimol), but the complete 228-canvas institutional scan
is CC BY-NC 4.0 and remains pixel-excluded. The single licensed composite is a
separate manuscript group from Wellcome MS Indic epsilon 1, but cannot by
itself establish the 200-line gate and still needs exact-original SHA-256 and
qualified line review. BDRC MW1NPL23 is separate from MW1NPL132; its
open/released scan has 25 unique IIIF canvases under BDRC's scanned-manuscript
public-domain policy. Its physical record assigns both ScriptNewa and
ScriptRanj without page-level attribution, so every text-bearing canvas must
be classified Ranjana/Newa/mixed/other by a qualified reviewer. The frozen r12
catalog stores no source pixels and reports zero qualified decisions, audited
lines, labels, OCR, training, or scoring. The r13 inventory audit passes with
11 candidates (5 Bhujimol, 6 Ranjana), 3 acquisition-ready manifests, 2
completed remote acquisitions, and zero admitted sources. Its tests reject
license, institutional-script, manifest-license, duplicate-service,
probe-hash, and line/admission drift. Evidence:
outputs/newari-second-manuscript-source-catalog-2026-07-15-r12/ and
outputs/newari-source-candidate-audit-2026-07-15-r13/.
The r14/r15 acquisition round executed the second-source handoff on
omacs-MacBook-Pro (68,719,476,736 memory bytes) without OCR, training, or
scoring. The exact Commons original and all 25 BDRC canvases decoded at their
frozen dimensions; all 26 byte hashes are unique and total 12,738,550 bytes
(3,435,984 Bhujimol; 9,302,566 Ranjana). The Commons SHA-256 is
938013af2c4c16ac84164a41f903cf34d131663b7a069e58df779bacfcd4258d and
its upstream SHA-1 also matches. Separate profile checksum manifests bind one
Bhujimol image and 25 Ranjana images, while the orchestration host mirrors
metadata only. The local audit creates 28 blank qualified-review units: three
Bhujimol panels (28r, 28v, 29r) and 25 BDRC canvases. All decisions,
audited lines, labels, admissions, and claims remain zero. Evidence:
outputs/newari-second-source-prediction-free-review-2026-07-15-r14/ and
outputs/newari-second-source-remote-acquisition-audit-2026-07-15-r15/.
The refreshed r16 central inventory now reports 11 candidates, 7 explicit
commercial-permission statements, 3 acquisition-ready manifests, 4 remotely
fetched-and-hash-recorded sources, and still zero admitted sources or support
claims (outputs/newari-source-candidate-audit-2026-07-15-r16/).
The r17 review-readiness round joins the four acquired manuscript groups
without copying pixels or inventing decisions. It binds 264 unique remote
source images to 266 blank review units: Bhujimol has 218 images/220 units
across Wellcome b14901560 and Cambridge MS Add.2137; Ranjana has 46
images/units across BDRC MW1NPL132 and mixed-script MW1NPL23. Both
profiles meet only the two-manuscript-group precondition. Qualified decisions,
estimated lines, audited lines, labels, admissions, OCR, training, scoring,
and claims remain zero, so the >=200-line gate is explicitly false for both.
Evidence:
outputs/newari-acquired-source-review-readiness-audit-2026-07-15-r17/ and
docs/handoffs/HANDOFF-newari-acquired-source-review.md.
The r43-r45 correction separates manuscript identity from exact transcription.
The Wellcome story identifies b14901560 / MS Indic epsilon 1 as the
Aṣṭasāhasrikā Prajñāpāramitā Sūtra, but its 217 IIIF canvas labels are all
-; no folio or line transcription accompanies the images. A GRETIL e-text
exists for the Vaidya 1960 edition, but its page says it is for reference only,
defers terms to the source file, and supplies no crosswalk to this manuscript.
It is therefore neither rights-cleared prefill nor ground truth. BDRC's frozen
physical record labels MW1NPL99 “Unidentified” and supplies no e-text.
Cambridge's official TEI says the Nāradasmṛti begins at verse 5.8cd on 28r,
but the manuscript interleaves that Sanskrit root with a Newari
commentary-cum-translation; the TEI explicitly states that the Newari legal
text and translation are unpublished and gives only short catalog excerpts.
Accordingly, the r45 fail-closed audit reports three routes checked, zero exact
line-aligned transcriptions, zero admissible prefills, and zero labels.
The same pass found that the frozen r15 worksheet named panel-28r,
panel-28v, and panel-29r but pointed all three rows at the same uncropped
2,332 x 1,800 four-folio Commons composite. The row identities and licensed
source hash remain valid, so no frozen review data was rewritten. The r43
profile dispatch and r44 combined UI instead render fixed source-bounded
regions [0,0,2332,400], [0,430,2332,420], and
[0,890,2332,410], crosschecked against institutional canvases 48–50. They
fail closed to the full original if dimensions or bounds drift, preserve the
full-image link and SHA-256, contain no OCR text or labels, and keep the same
339-row assignment scope. The complete selected regions were visually checked
against the CC BY-SA Commons original. Tests cover rendering metadata,
source-dimension refusal, immutable review scopes, output checksums, and the
companion-text admission gate. Artifacts:
outputs/newari-qualified-page-review-dispatch-2026-07-15-r43/,
outputs/newari-qualified-page-review-ui-2026-07-15-r44/, and
outputs/newari-bhujimol-companion-text-audit-2026-07-15-r45/.
The r41 Devanagari-Newari source round found one exact official State Portal
record titled SIKKIM HERALD-NEWA EDITION, dated 13 April 2022, under the
permission-covered sikkim.gov.in/uploads/SikkimHerald/ prefix. The existing
Sikkim source policy remains exact at SHA-256
2247033ac1ce7de8d99cad22feb702f9fe48a11341a2fb2d20c1d6d48b06717a.
An eight-byte HTTP Range probe confirms a PDF response with a total size of
3,975,535 bytes; no PDF body or page image was downloaded locally. The newer
IPR portal currently returns zero Newar records and was used only as negative
discovery evidence. The administration URL currently exposes only the official
Government of Sikkim site shell, so no substantive publication claim is derived
from it. The target writing-system route is Devanagari, but the edition's actual
source script, fonts, and third-party content remain explicitly unconfirmed
until all pages are rendered and reviewed on an approved remote host. The
catalog therefore admits zero pixels, decisions, labels, real lines, models,
or metrics. Evidence:
outputs/newari-devanagari-sikkim-source-catalog-2026-07-15-r41/.
Two tempting Commons catalog leads were rejected for production use in the same
round. AN-PM-12.pdf and AN-PM-09-Dvadasa Tirtha Yagu Bakham.pdf are described
on Commons as old Newari material in Devanagari with Public Domain Mark labels,
but the British Library EAP record for AN-PM-09 states that access is for
research purposes only. A downstream Commons rights label is not treated as
authority to erase that source restriction. Recent uploader-applied Internet
Archive CC0 metadata was likewise kept out when uploader authority could not be
established.
Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded
2026-06-18 scorecard in docs/RECOGNIZER_GOAL_PROGRESS.md):
- Limbu/Sirijonga 0.041 real-print (champion:
runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23, cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval). - 8 Devanagari languages 0.075–0.129 real-print (Majhi, Magar, Sunuwar-Deva, Magahi, Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer.
- Sunuwar (Jenticha) 0.121 real-crop after the same-font retrain (0.342 synth-only).
- Tirhuta 0.343 and Newa 0.613 synthetic-only — weak, transliteration-fed, honest fails; Newa needs real printed-register Nepal Lipi, not more pixels.
- Kirat Rai (70 real crops) and Lepcha (132 real crops) blocked on legacy-font converter coverage outside their eval editions. Gurung Khema now has official real-print source material: two 2022 State Portal Herald editions yield 749 mechanically convertible line candidates, but zero rows are admitted pending independent provenance review, native pixel-label review, and split/leakage gates.
The near-term pack claim should therefore be script-family shaped: one Sirijonga recognizer, one shared Devanagari recognizer covering the many Devanagari-written languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa, Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval crops exist.
Focus order (2026-07-04 steer): Limbu first, Newar second. The 2026-07-15
Newari steer requires Newar/Prachalit, Bhujimol, Ranjana, and Devanagari as
separate writing-system profiles. Newar state: the
Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own
synthetic eval; transliteration-fed training failed; real printed-register Nepal
Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar
route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20
Gorkhapatra pages; newar added to the registry (Devanagari primary + Prachalit
secondary; audit passes); a frozen by-date pack built at
outputs/newar-devanagari-silver-pack-v1/ (train 4,096 / dev 1,528 / test 1,139
lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently
verified; pack sha256
1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85); and the
first real-print Devanagari-Newar baseline was scored locally (CPU export
route, tools/score_recognizer_pack.py, per docs/LOCAL_RECOGNIZER_SCORING.md):
the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at
codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158 (n=1,139 silver
lines — machine-converted labels, diagnostic, not claim-grade). The same-day
warm-start fine-tune (newar-ft-v1: deva-v2 base, LR 2e-5 cosine, 60 epochs,
batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance
ashish-experiments, ~68 min; artifacts + train log at
outputs/aws-newar-deva-ft-v1/) scores the same frozen test at codepoint CER
0.0465, grapheme CER 0.0955, exact-line 0.245 — a 25/27% relative CER drop and
+55% exact-line over the deva-v2 baseline, scored through the identical local
CPU path. Newar's grapheme CER now sits inside deva-v2's own-language range
(0.075–0.129): the second focus language reached Devanagari-pack parity from
zero in one day. Same claim boundary: silver machine-converted labels,
diagnostic, not claim-grade until native review. Next Newar lever: native
Prachalit (real printed-register data), and page-level Newar eval for the
region-filtering track.
Source Rule
For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata
only. Use https://gorkhapatraonline.com/publications/naya-nepal to find the language,
date, and article, but do not use the blog/card image for OCR, training, or evaluation
when the full paper exists. The authoritative source is the matching Gorkhapatra epaper
PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus
the text layer extracted from that same PDF page, with PDF URL/hash, page number,
rendered image hash, extraction command/version, font evidence, and converter/map hash
recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with
src/ocrtech/limbu_legacy.py + data/mappings/limbu-legacy/Limbu.map. The same
pattern — classify the text layer, then build an explicit converter only when the font
requires it — replicates to every other language. Manual transcription is the fallback
for no usable text layer, not the default.
Getting More Gorkhapatra Pages (the specified harvest recipe)
All numbers below were verified live on 2026-07-04.
Archive size. The Naya Nepal publication archive paginates at
https://gorkhapatraonline.com/publications/naya-nepal?page=N with 5 articles per
page and runs to page 352 before drying up: **1,750 language-page articles total**
(about 2 per day going back years). The 60-page crawl already yielded 300 items across
53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly
older dates.
Epaper endpoint behavior (the critical constraint). The date-range endpoint
https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra
caps results at 30 PDFs per query and ignores page=N pagination. A wide window
silently drops most dates — the 2026-07-04 six-window discovery pass aligned only
111/300 items for exactly this reason, with unmatched dates interleaved among matched
ones. Seven-day windows return complete daily coverage (7/7 verified). Always
harvest with weekly windows.
The recipe (one command; add one --epaper-url per week to cover the publication
date span being harvested):
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
ocrtech audit-gorkhapatra-language-pages \
--out outputs/gorkhapatra-language-pages-full-<date> \
--publication-url https://gorkhapatraonline.com/publications/naya-nepal \
--publication-pages 80 \
--epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \
--epaper-url '...one per week, going back...' \
--download-pdfs \
--render-pages \
--timeout-seconds 180
The command aligns every publication card (all languages at once — alignment is
language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL),
searches pages for the language-page markers (भाषा पृष्ठ and its legacy-encoding
form efiff k[i7), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly
windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with
400 publication items, 195 aligned dates, 394 rendered language-page hits, zero
warnings at outputs/gorkhapatra-language-pages-full-2026-07-04/. Batch 2
(2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more
items: 798 items, 294 aligned dates, 589 further page hits, at
outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/. The merged normalized
inventory (tools/build_gorkhapatra_language_pack_inventory.py) shows 56
languages, 983 rendered language-page hits, most languages with ~20–27 pages
spanning 14 months, at outputs/language-pack-inventory-merged-2026-07-04/. For
comparison, the entire previous state of the project had 8 usable pages for one
language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least
March 2024 — verified the epaper endpoint serves complete weeks there) is another
~2× and stays month-batchable with the same recipe.
Scaling to the full archive. Full depth is ~700+ unique dates ≈ 35 GB of PDFs.
Do not do it in one run: batch by month (4–5 weekly windows + the matching
publication-page depth), run extract-gorkhapatra-language-page-pdf-text per batch,
then keep rendered pages, extracted text, and hashes; PDFs can be pruned after
extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume
(cdjk@100.110.148.43:/Volumes/TRANSCEND) is the right home for a full-depth mirror.
Per-language downstream. After harvest, run per language:
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
ocrtech extract-gorkhapatra-language-page-pdf-text \
<pack-manifest.jsonl> \
--candidate-language <label> \
--target-font <font> \
--out outputs/<language>-pdf-text-<date>
then classify each page's text layer (native Unicode / legacy font / Devanagari transcription / Latin / none) and only then decide converter vs manual transcription.
Font/text-layer facts already established (30-page, 15-language extraction at
outputs/gorkhapatra-multilingual-language-pages-current-2026-06-24-pdf-text/):
- The body font of non-Limbu language pages is
nayanepal— a legacy 8-bit Preeti-family encoding, not Unicode (8,845 of 11,791 spans; the text layer readsk[i7 ;+of]hswhere the page shows पृष्ठ संयोजक). The converter already exists:nepal_ttf2utf.convert_devanagari(font="nayanepal")in the sibling repo/Users/cdjk/github/llm/nepal-ttf2utf(Preeti base map + nayanepal extension glyphs + NFC, ~97–99% clean on real Gorkhapatra pages), already used bytools/build_gorkhapatra_devanagari_real.pyfor the 79k-row real Devanagari corpus. The remaining leverage is wiring it per language: run it over each harvested language page and emit per-language silver line packs with the same faithfulness gate (clean-Unicode allow-set,conversion_gapaccounting, leakage checks) that the Devanagari pack builder already implements. The same sibling repo also hosts the limbu, kiratrai, lepcha, and sunuwar legacy converters. - Tibetan-script fonts (
TibetanMachine,MonlamUniOuChan5,Qomolangma-*) appear on Mugal and related pages, but every such span has empty extracted text (verified 2026-07-04 across all 2026-06-24 samples) — Tibetan-script pages are visual-only, like the known Limbu2082-11-12date. They need OCR bootstrap or manual transcription; no converter is possible. Gorkhapatra-*,Nagarik-*,TimesNewRoman*,MyriadPro-*are masthead/furniture fonts — useful as non-target filters for target-region selection.NamdhinggoSILLappears only on Limbu dates (the existing Limbu path).
What We Built (Limbu template, still true)
The Gorkhapatra alignment path: publication-page discovery, epaper-date matching, PDF
download/rendering, page-marker search, PDF-native span/font extraction, review
sheets, reference templates, and dashboard visualization. The expanded Limbu harvest
has 9 discovered dates, 8 usable PDF-text-layer pages, 2,441 line rows, 2,430 usable
silver labels (outputs/limbu-gorkhapatra-pdf-backed-silver-2026-07-03-v1/).
The Limbu recognizer training/eval path: 48,507-row combined pack; real-heavy fine-tune champion at codepoint CER 0.011336 / grapheme 0.024736 / exact-line 0.729 on the frozen 255-line silver eval. Synthetic hardening lessons: targeted+Unicode packs regressed; v3 dictionary warm-start catastrophically failed (~0.98 CER); keep the real-heavy v2 champion; future gains come from real/book-domain harvesting or a deliberate rehead, not blind synth rows.
The page/book OCR loop: local + HF Space (spaces/limbu-ocr/, deployed private at
https://huggingface.co/spaces/ampixa/limbu-ocr) with line OCR, Paddle Page OCR,
Paddle Book OCR, image-line fallback, runtime checks, annotated pages, and crops.
Page-level diagnostics on the 8-page silver pack: Rows LTR + y/conf70 filter scores page cp CER 0.414330, line F1 0.963009, matched-line cp CER 0.026039, order accuracy 0.993176. Recognition on matched lines is strong; full-page CER is dominated by page reconstruction, target-region selection, and furniture filtering — that is the pipeline problem to solve, and it is language-independent.
Supervised line-detector infrastructure from CDC PDF geometry (287 pages, 26,160 annotations, mAP50 ≈ 0.921 on PDF-rendered val) — evidence the detector path works, not a solved product.
A claim-grade eval harness (versioned packs, hashes, provenance, normalization profiles, cp/grapheme CER, CIs, paired comparisons, leakage manifests) — parked as a guardrail; do not let it eat work blocks that should go to the pipeline.
Current Working State
The strongest component is the Sirijonga line recognizer. The weakest component is
end-to-end page/book reconstruction: target-region selection, furniture filtering,
line preservation, and ordering. Of the three Limbu-locked code spots, two were
generalized on 2026-07-04 (Codex worker, tests green): _script_ratio in
src/ocrtech/engines.py is now driven by the shared SCRIPT_CODEPOINT_RANGES table
covering all pack scripts (devanagari, limbu/sirijonga, latin, sunuwar/jenticha,
lepcha, kirat_rai, newa/prachalit, tirhuta, gurung_khema, ol_chiki, tibetan, with
aliases), and the image-line filter in src/ocrtech/image_lines.py accepts any
script key from that same table. The third spot fell the same day: the
src/ocrtech/limbu_runtime.py preflight now takes required_script (default
limbu, validated against the shared table; CLI --required-script), so a router
card for any pack script can be preflighted. The engine classes themselves (CandidateEngine, PaddleOcrEngine,
ScriptSelectCompositeEngine, build_image_line_document, structure.py reading
order) are already parameterized — the rest of the generalization is registry-driven
wiring, not a rewrite.
The Paddle Pipeline (whole pipeline, not just recognizers)
Target architecture, all stages Paddle-first and language-pack-parameterized by the
per-language pipeline config configs/nepal-language-pack-pipeline-v1.json
(50 languages: label, registry id, scripts, recognizer artifact, dictionary,
converter, text-layer class, harvest counts — generated 2026-07-04 from the
inventory):
- Capture/prep — existing
prepare-limbu-capturepath, already language-agnostic in behavior; rename/rescope. - Detection — Paddle text detection as production default; YOLO line detector and morphology image-line boxes stay as diagnostics/rescue passes.
- Script identification & routing — generalize
_script_ratioto a Unicode-block-driven scorer over the pack's script list (Devanagari, Limbu, Jenticha/Sunuwar, Lepcha, Kirat Rai, Newa, Tirhuta, Gurung Khema, Ol Chiki, Tibetan, Latin), then route each line to the script's recognizer from the registry. - Recognition — per-script PP-OCRv5 recognizers (existing runs) with per-script dictionaries and input-width profiles (Sirijonga width-480 lesson).
- Target-region & furniture filtering — non-oracle region selection using layout plus font/language signals; furniture fonts and masthead geometry as negative evidence. This is the main open research item; score against the Rows LTR + y/conf70 baseline.
- Reading order & assembly —
auto_layoutselection (wide-column RTL vs rows) per page, conservative assembly stays diagnostic. - Export — Unicode text, JSON with provenance, Markdown; review queues + dashboards per page (existing Limbu artifact shapes, renamed pack-wide).
- Eval guardrail — frozen real-page packs per language: page CER/WER, line P/R/F1, reading order, blank/extra/missed lines, review burden.
PP-StructureV3 and PaddleOCR-VL remain the documented fallback/teacher direction for
hard layouts, per docs/NEPAL_MULTILINGUAL_OCR_GOAL.md.
Production book-ingestion plan (2026-07-04)
The production target: a book (scanned PDF, image folder, phone captures) in; editable Unicode, provenance JSON, and Markdown out; a review queue covering only lines that need a human; frozen regression packs gating every model/config change. The production KPI is review burden (can an editor fix a book in hours), not CER alone. Phases in leverage order:
PP-StructureV3 spike — DONE 2026-07-05, verdict: stays fallback. The 2026-07-04 CPU run died producing nothing (~6 h on page 1/8); the re-run on the L4 GPU did all 16 rebuilt pages in ~4 s/page. Scored on the rebuilt 16-page / 4,895-line Limbu silver pack, same champion recognizer and same
run_page_ocr_eval.pyflags in every arm (diagnostic, non-claim):arm page cp CER det F1 matched cp CER order acc PP-StructureV3 raw (its own reading order) 0.797 0.904 0.0345 0.778 PP-StructureV3 boxes + Rows-LTR + y/conf70 0.421 0.934 0.0331 0.993 Current stack: Paddle det + Rows-LTR 0.437 0.942 0.0266 0.993 Current stack + y/conf70 filter (baseline) 0.413 0.966 0.0266 0.993 Stock PP-StructureV3 loses on every axis even after grafting our reading order and filter onto its boxes: worse detection, worse crops for our recognizer (matched CER 0.033 vs 0.027), far worse native reading order, and it needs a GPU. The Paddle det + Rows-LTR + y/conf filter chain stays the production backbone; PP-StructureV3 remains the documented fallback / teacher for table- and figure-heavy layouts (not exercised by these text-heavy newspaper pages). Artifacts:
outputs/ppstructure-spike-rebuild-2026-07-05/(stage-1 GPU JSONs, stage-2 documents, all score dirs); baselineoutputs/limbu-baseline-rebuild-2026-07-05/.Detector fine-tune — cash in the supervised det goldmine: 983 language pages with PDF-text-layer line geometry across 50+ languages plus 26k CDC annotations; fine-tune PP-OCRv5 det to fix the center-column miss pattern.
One config-driven book runner — generalize
run_limbu_book_ocr.pyinto a language-packrun_book_ocr(input → prep → layout → det → script-route → recognize → assemble → export + review queue), resumable per page.Acceptance gates — frozen page packs (Limbu 8-page, Newar pages, one real book) with thresholds on page CER, line F1, reading order, review-burden %; the parked eval harness serves as the gate runner, not a research project.
Packaging/serving — versioned per-script model bundles, Space/API on the pack pipeline, CPU eval path, GPU batch for book jobs.
External Native-Script Real-Data Harvest (2026-07-06)
Rebuilt the lost newa-zenodo-6967421-v1 asset and swept Zenodo/HuggingFace/
GitHub/Internet Archive for other real (non-synthetic) native-script OCR
material. Everything landed at
data/external-language-resources/native-script-real-2026-07-06/<script>/
with a provenance.json per source (license, byte size, md5, verify method).
Full details in that tree; headline results:
- Newa/Pracalit (Zenodo 6967421, CC-BY-4.0): re-downloaded, md5-verified
(
76d464a6b656482b7c0594c0cb37fb18), extracted. Correction to the record: it is 441 page images + 441 PAGE-XML, not the 683 pages this doc previously cited — 441 matches the record's ownmetadata.xmland the Zenodo description's "441 training pages"; the "242 validation pages" the same description mentions are not in this file. Do not carry 683 forward. Mirrored bit-for-bit at heiDATA (10.11588/DATA/WI9184, same title/size) — useful as a second-source integrity check, not new content.tools/newa_zenodo_build.pyexpectsdata/newa-zenodo/extracted/; this pull is at the new path above, so point the tool there (or symlink) before re-running the line-crop pipeline. - Newa/Pracalit, other leads that did NOT pan out: a newer O'Neill paper
(Zenodo 17268040, "Revisiting HTR for Nepalese Manuscripts in Pracalit
Script," 2025) cites no new public GT beyond 6967421. A published
"First-Ever Prachalit Newa Lipi Character Dataset" paper exists, but its
data (via
sobit-nep/Prachalit-Script-Character-Recognition-using-Deep-CNNon GitHub) is a dead Google Drive link (404, not actually "anyone with the link") — verified, not assumed. heiDATA'sFID4SA-GTdataverse (the same group that hosts the Newa mirror) also has Devanagari, Malayalam, and Bengali printed GT sets, CC-BY/CC0 — none of them our target scripts, not pulled. - Ol Chiki (Santali): no image+GT dataset exists on Zenodo/HF/GitHub (only
a Tesseract
.traineddatablob and app scaffolding with no bundled data). Internet Archive had real wins instead:chiki-chemed(a Santali primer, real scanned pages, CC BY-NC-ND 4.0 — note the NC-ND clause before any commercial use) andaale-chhatka-pdf-e-magazine-{2023,september-2023}(a real monthly Ol-Chiki literary e-zine, self-published by its own editor's archive.org account; the 2023 item carries the Creative Commons Public Domain Mark 1.0, which is neither CC0 nor a license, while the September item has no license field despite the same publisher — don't assume terms transfer). All three downloaded, size-verified, script-confirmed by rendering a sample page. - Tirhuta/Mithilakshar — best new lead of this pass: the Maithili e-zine
Videha (world's first Maithili webzine) publishes a parallel Tirhuta
edition of every issue, archived at Internet Archive under the
videha_*family, CC BY-SA 4.0, issues 001–149 confirmed present (3 items, ~12 GB) plus larger combined items (videha-262~23 GB,videha-306~40 GB) referencing further issues — not bulk-downloaded, catalogued only. Pulled one sample PDF and confirmed by rendering it that the pages are genuine Tirhuta-shaped glyphs, butpdftotexton the embedded text layer yields Devanagari-range codepoints, not the true Tirhuta Unicode block (U+114xx). That means these are legacy glyph-substitution fonts — the same shape of problem this repo already solved for Gorkhapatra Devanagari (nayanepal/legacy Preeti-family). If a Videha-specific font map can be built, this is real running text at real periodical scale, the same playbook as the Limbu win. This is the single most promising unexploited source found this session. - Lepcha / Kirat Rai / Sunuwar / Limbu (Sikkim Herald): the four newer IPR
listing endpoints (
?language=Rai|Mukhia|Lepcha|Limboo) remain useful for discovery, and no Zenodo/HF/GitHub/IA periodical archive was found. This 2026-07-06 note is superseded for admission by the 2026-07-15 rights audit: the express reproduction permission covers material hosted on the official State Portal, not automaticallyipr.sikkim.gov.in. The IPR tool is now metadata-only by default; admit only a mirroredsikkim.gov.in/uploads/SikkimHerald/file or separately permitted IPR file. - Nepal Lipi periodicals: the Internet Archive
nepalbhasatextscollection (477 items) is almost entirely Devanagari-script Nepal Bhasa print media (Situ, Jhee, Thaunkanhe, Khonapau, etc. — already the domain of the existing Newar-Devanagari track, not this one). Checked several candidates that looked native-script by title:kmc-notice-2023is a single decorative Ranjana logo clipping, not running text.NepalLipiSangrahaByHemrajShakyavamsha_201712(no stated license) is a Devanagari-language book about Nepal Lipi that does contain real alphabet-plate specimens of historic native scripts (Kunmol, etc., with transliteration) — a glyph reference, not periodical running text. No genuine Nepal-Lipi-script periodical PDF archive was found.
Recommendation, one line per script: Newa — nothing left to pull, move to
building the line-crop pipeline on the corrected 441-page set. Ol Chiki — use
aale-chhatka-pdf-e-magazine-2023 (publisher/uploader Public Domain Mark 1.0,
confirmed script) as the first real
training/eval seed. Tirhuta — decode the Videha legacy Tirhuta font (highest
leverage find this session: real, licensed, periodical-scale) before anything
else. Lepcha/Kirat Rai/Sunuwar/Limbu — use the IPR listing only to discover
editions, then catalog hash-pinned mirrors on the permission-covered State
Portal; do not bulk-download IPR PDF bodies without separate exact-prefix
permission evidence.
Native-script wave 1 results (2026-07-06)
- Ol Chiki v1 (first recognizer for the script here): PP-OCRv5 mobile
from generic pretrained, 56,640 synth lines (Santali Wikipedia corpus ×
3 Noto weights × 6 degradations), 25 epochs. Synth eval best exact-line
0.546 / norm_edit_dis 0.800. Real print (234 Aale Chhatka magazine
lines via the new OLCKOptimum converter): cp CER 0.2245, exact-line
0.184 — real-print performance ≈ synth-eval performance, so the
synthesis recipe transfers; quality itself is the v1 limit. Known levers,
in leverage order: (1) same-font synth — we hold the extracted
OLCKOptimum TTFs; replay the Sunuwar same-font lesson (0.342→0.121);
(2) resolve the 2 uncertain converter bytes (
n,T) to unlock 653 more real lines for train/eval; (3) real-line fine-tune on an issue-split of those lines. Artifacts:outputs/olchiki-v1-run-2026-07-06/(checkpoint+export+scores), packdata/olchiki-real-eval-pack/, converternepal-ttf2utf@acab167. Claim note: Santali OCR exists elsewhere (India); this is pack coverage, not a first-ever claim. v2 (same-font, 2026-07-07): re-rendered the same corpus with the OLCKOptimum fonts extracted from the magazine PDFs (all 48/48 block chars covered by the embedded subsets) + one Noto weight: real-print cp CER 0.1911, exact-line 0.321 (same 234-line pack) vs v1 0.2245 / 0.184 — the same-font lever works on this script too, though less dramatically than Sunuwar's; remaining error is heading/display type, tight-leading crop bleed, and genuine model limits. Artifacts:outputs/olchiki-v2-run-2026-07-07/. A later asset-independence audit found that those fonts came from the exact evaluation edition, so v2 remains a useful diagnostic but cannot provide an edition-independent claim result. - Newa native v1 (manuscript domain): honest failure. 150 epochs on the
recovered Zenodo 441-page GT (2,391 train lines, by-manuscript split,
800px width, PP-OCRv5 mobile from generic print pretrained): dev exact
0.0016 / norm_edit 0.144, and crucially train-side norm_edit is also
0.146 — the model could not fit its own training set, so this is a
recipe failure (print-pretrained mobile backbone vs dense 16th–19th-c.
handwriting at 63–154 chars/line), not a hand-generalization failure.
The reference point (O'Neill 2022/2025, Transkribus HTR on the same data)
shows the task is learnable with HTR-specific training. v2 design:
synth-first curriculum — render the TRAIN-split transcription text (plus
the 36k-char web corpus; no test text, no transliteration) in the 2
Unicode Newa fonts with degradations, pretrain, then manuscript
fine-tune; consider the server rec backbone. Artifacts:
outputs/newa-native-v1-run-2026-07-07/.
Fan-out results (2026-07-13, Codex wave)
- Ol Chiki uncertain bytes RESOLVED (
nepal-ttf2utf@5822f30): bytesn→ᱱ (EN U+1C71) andT→ᱛ (AT U+1C5B), each confirmed by three independent evidence lines (glyph stroke features, corpus context distributions vs the 204k-line Wikipedia corpus, lexicon hit-rates 0.89 vs 0.57–0.60 runner-up); evidence atoutputs/olchiki-uncertain-bytes-2026-07-13/. Clean real-print yield rose 234 → 895 lines; expanded pack atdata/olchiki-real-eval-pack-v2/. olchiki-v2 on the 895-line pack: cp CER 0.1275, exact-line 0.197 (scored on omac, 5m18s) — the new body-text-dominated benchmark reads meaningfully better than the display-type-heavy 234-line pack suggested (0.1911). Pack v2 was initially treated as the reference eval, with 895 candidate lines. The following audit superseded that interpretation before any claim use. Corrections (2026-07-14, license audit + label audit): (a) public claims may use only the publisher/uploader public-domain-marked issue; the mark is not CC0 or a license; (b) the label audit found 67/895 crops blank (pack-builder bug, all in that issue). The resulting cp CER 0.1018 / exact 0.191 (n=366) and combined clean 0.0826 (n=828, internal) are historical diagnostics, not claim headlines: surviving crop/label mismatches, ambiguous punctuation, incomplete provenance, and same-edition font leakage remain. Also found: the Newar-Devanagari packs carry systematic converter whitespace artifacts (spurious space after halant / before combining marks; ≥1 in ~45% of test lines) — word content is faithful, but a whitespace-normalization pass is required before those numbers are claim-grade. See docs/DATA_LICENSING.md and the verification-wave section.
Newa root cause found (2026-07-15): CTC timestep cap, not data or height
The height-ablation prep (dry-runs on omac; artifacts at
outputs/newa-height-ablation-prep-2026-07-14/, VERDICT.md) found the
mechanical cause of the v1 training failure: PPLCNetV3's train-mode forward
ends in adaptive_avg_pool2d([1,40]), hard-capping the CTC sequence at
T=40 timesteps at any input height or width — and 95.7% of manuscript
train lines exceed 40 chars (median 63, max 154). CTC cannot emit more
labels than timesteps: v1 could never fit its own training set, and a pure
height ablation is predicted to reproduce the failure. This coheres with
every other result: Limbu/Ol Chiki/Newar-Devanagari lines mostly fit ≤40
chars and their models trained fine. Also verified: heights 96/128 train as
pure config changes (3 finite loss steps each), but the EVAL path breaks
(Im2Seq asserts H==1) — a verified 1-line patch ships in the prep dir;
pretrained weights load identically at all heights; an earlier T=40 memory
estimate proposed L4 batches 16 (h96) / 12 (h128), but T=200 requires a fresh
approved-GPU measurement. The corrected Newa experiment therefore needs a single
train/eval pool width of T=200 (the exact CTC lower-bound maximum is 176 after
normalization), h96, and only the CC-BY-4.0 Zenodo manuscript source. A
backward-compatible pool patch has now been shape-tested on omac: defaults
remain train T=40 / eval T=100, while the opt-in h48 and h96 paths both produce
T=200 and a length-162 CTC loss is finite. The pre-existing zero-byte
newa_real_000041.png was not dropped: it was restored from the authoritative
local master and pinned at SHA-256
c86f18e93b123a87e26682b973c5db8a2f8931434f11f6c88a63105161920360.
Native-script verification round (2026-07-15)
- Newa CTC200 payload staged, not trained. Labels are narrowly normalized
(NFC, NBSP to space, whitespace runs collapsed, boundaries trimmed), and
every referenced crop decodes with ink. Exact-label and 16-codepoint
substring isolation removed 430 of 2,391 training rows, leaving 1,961.
Hitopadesa is the 144-line checkpoint-selection dev manuscript; the 635-line
Cambridge manuscript is sealed as
claim_evaland must not enter training or model selection. Post-filter overlap is zero and all labels are feasible at T=200. A prediction-free all-row Cambridge review bundle has now been built onomacfrom the exact payload and original Zenodo export: 635/635 bundled PNGs are byte-identical to their payload crops, all 635 decision rows remain blank, and the archive is explicitly not claim evidence. The crop labels still need documented native pixel review; any corrections require a fresh post-review exact/16-codepoint leakage audit before a claim-eligible freeze. A fail-closed freezer now revalidates the exact payload/source/review archives, immutable fields, pixels, all 635 decisions, normalization, and the >=200-row dedup gate. It keeps non-attested outputs claim-false and requires exact CTC200, train/dev, post-review leakage, checkpoint, and training-history bindings for claim readiness. The real blank worksheet fails atomically; no reviewed pack exists. GPU training remains approval-gated. Artifacts:outputs/newa-ctc200-prep-2026-07-15/,outputs/newa-cambridge-review-v1/, anddocs/handoffs/HANDOFF-newa-ctc200.md. - Newa balanced sampling is now a deterministic, disabled candidate. The
exact 1,961-row train-only capsule contains 146,153 Newa codepoint tokens
across 69 types and 87,777 extended-grapheme tokens across 585 types. The
schedule excludes the 2 codepoint and 71 grapheme singleton types, adds 490
references under a three-occurrence row cap, and preserves exactly the same
1,961 unique image/text pairs. Train/Hitopadesa exact-label and 16-codepoint
overlap are zero before and after scheduling; no Hitopadesa or Cambridge row
is copied. The six-file artifact reproduced byte-identically when rebuilt
with the pinned
regex==2026.5.9segmentation runtime. This is a sampling heuristic, not accuracy evidence, and no current payload or config enables it. Activation requires a newly hash-bound training path, approved GPU ablation, and the final corrected-Cambridge leakage audit. Artifact:outputs/newa-balanced-sampling-v1/. - Stock RepSVTR is a runnable CTC200 fallback, not a trained result. A
no-image, batch-1 CPU probe on
omacused the clean PaddleOCR v3.7.0 checkout and the official hash-pinned RepSVTR checkpoint. At H96xW1600, the train-mode CTC branch and eval-mode full forward both emit T=200, leaving 24 timesteps above the audited corpus maximum of 176; an alternating length-176 synthetic CTC loss is finite. All 301 backbone tensors and 394/398 total tensors are pretrained-compatible, with only the four Newa-vocabulary-sized tensors reinitialized. A follow-up synthetic CPU probe then exercised the exact five-tensor loader contract through one full train-mode MultiHead forward, CTC+NRTR MultiLoss, backward, and scheduler-positioned AdamW update. It observed CTC[1,200,108], NRTR[1,177,112], finite component losses, 279/279 finite nonzero gradient tensors, and finite positive updates in the backbone plus both heads; peak RSS was 1,646,444,544 bytes. No image, label file, dataset row, prediction, score, GPU, cloud, or network action occurred. The exact configured warmup starts at LR 0, so the probe explicitly advanced the scheduler 1,961 times without optimizer calls before its sole update at LR4e-5; this is mechanism evidence, not first-iteration chronology or training-loop history. Together the probes establish an independent backbone/temporal path from patched PPLCNetV3 and close the synthetic full-loss/backward/update gap, not the framework, real-data, GPU-memory, convergence, or accuracy gaps. Training remains approval-gated, and selecting this path requires the distinct stock-RepSVTR claim-attestation profile now implemented intools/audit_newa_cambridge_claim_readiness.pyand its freezer. That profile pins the exact config, feasibility report, fully clean PaddleOCR checkout, stock sources, official initialization, approval, run log, selection record, and a CPU-loaded final state dict with the exact 398 Newa model keys/shapes. The freezer independently requires the raw approval/run/log/selection records and the selected checkpoint. The legacy four-file PPLCNetV3 profile remains unchanged. Identity and execution authenticity are still an external custody/signature boundary; no real approval, training record, compatible final checkpoint, attestation, or score exists. Artifacts:outputs/newa-repsvtr-ctc200-feasibility-v1/andoutputs/newa-repsvtr-full-train-step-v1/. - Ol Chiki v2 withdrawn from claim use; v3 is only a review candidate. A
full audit found blank/rule-only crops, unrelated or neighboring text,
unresolved legacy punctuation, a stale converter-commit claim, and training
fonts extracted from the evaluation edition. The public-source-only v3
builder pins the PDF and resolved map, uses neighbor-clamped vertical crops,
rejects blank renders, emits line-level hashes/provenance, and refuses to call
its output frozen or claim-eligible. The remote candidate contains 358 rows
after 75 blank renders were excluded. Every row still requires review by an
Ol Chiki reader; no new score is permitted before freeze and independent
exact/substring and asset-isolation audits. A fail-closed freezer validates
the pinned candidate and all-row review archives, the source PDFs, every crop
and provenance binding, and all 358 immutable review decisions; it refuses
claim eligibility without separately bound leakage and asset-isolation
attestations. No decisions or frozen pack exist. Artifacts:
outputs/olchiki-claim-eval-v3-2026-07-15/. - Tirhuta now has an edition-disjoint train/eval candidate route, not a
score. PyMuPDF retained glyph IDs for all 3,306 U+FFFD characters in the
152-page issue-001 sample. A fail-closed 164-entry functional map recovered
every one; 30,396/30,581 Janaki trace fragments then converted cleanly
(99.395049%). Issue 001 now supplies 400 nonblank real-line eval candidates
across 116 pages and a prediction-free all-row review packet. The separately
dated, CC BY-SA 4.0 April 2008 issue required 34 additional GID mappings.
Exhaustive shaping with the hash-pinned Janaki 1.000 reference font
reproduced all 34 plus the prior 164 with zero regressions. Its 2,000-row
candidate has 2,000 unique image hashes and zero image-audit issues; removing
one 16-codepoint cross-issue substring conflict leaves 1,999 training rows
with zero exact or long-substring overlap. Thirty new GIDs have clean crop
witnesses and four have shape evidence only. The 400 eval rows and 30 focused
mapping rows remain native-review pending; no model or CER exists. Artifacts:
outputs/tirhuta-videha-gid-recovery-v1/,outputs/tirhuta-videha-line-candidate-v1/, andoutputs/tirhuta-videha-april2008-train-candidate-v1/. A separate fail-closed freezer now validates the two pinned issue-001 archives, source PDF, packaged audits, converter/runtime provenance, and the exact crop bytes shown to reviewers. It requires all 400 immutable decisions, at least 200 nonblank unique accepted lines, and Tirhuta-bearing human labels. Claim eligibility remains false unless an independent attestation binds the reviewed label-file hash and the exact April train label/manifest hashes and passes exact-text, 16-codepoint substring, edition, and model-selection isolation. The decisions and attestation do not yet exist, so no pack was frozen and no score was run. - The other-script registry was downgraded to extant evidence. The Kirat
Rai pack has 70 rows / 53 unique labels, and all 53 occur in its real-render
training inventory. The shared Devanagari harvest has 79,364 labels but only
40,358 images, so it is a training inventory with 39,006 missing assets, not
eval. Sunuwar's historical 429-row pack/checkpoint and Lepcha's source PDFs
and checkpoint are absent. Historical metrics remain historical and none of
these routes is a current script delivery. Registry evidence:
configs/nepal-language-pack-pipeline-v1.json,configs/recognizer-eval-pack-normalization-v1.json, andconfigs/eval-harness-recipes-v1.json. - Sikkim Herald source permission was found and the three routes reopened.
The earlier Indian-government-work term analysis missed the Government of
Sikkim State Portal's express copyright permission. Its official copyright
page permits free reproduction in any format/media without specific
permission when the material is reproduced accurately, is not used
derogatorily or misleadingly, and the source is prominently acknowledged;
explicitly identified third-party material is excluded. The same portal
hosts Sikkim Herald, and the IPR Department identifies it as its official
government paper. A live metadata-only enumeration of the newer IPR portal
found 19 current Lepcha, 22 Mukhia/Sunuwar, and 5 Rai discovery candidates,
while the State Portal exposes dated editions for all three. A bounded live
State Portal catalog retained 29 unique canonical PDF URLs: 8 Lepcha, 11
Mukhia/Sunuwar, and 10 Kirat Rai. It rejected one duplicate and one non-PDF
path. The searches advertise 193/167/174 records, but
PageNo=1/2returned identical URL lists, so the artifact is explicitly incomplete. It fetched no full PDF body and made only three eight-byte range probes against samples already full-hash-pinned by the prior policy. Every edition remains third-party-screening pending, train/eval blocked, and claim-grade false. Because the permission page is domain-specific, onlysikkim.gov.in-hosted PDFs are catalog candidates; IPR-subdomain files remain discovery-only unless mirrored or separately permitted. This resolves the State Portal source-permission gate under a custom permission, not a public-domain claim. It does not restore the lost packs/models or cure the Kirat Rai pack's 100% text leakage. Next steps are full-hash acquisition and third-party screening of selected editions, followed by fresh edition-disjoint converter and native label audits. Evidence:outputs/sikkim-herald-source-rights-2026-07-15/,outputs/sikkim-herald-state-portal-catalog-2026-07-15/,docs/DATA_LICENSING.md, anddocs/ATTRIBUTIONS.md. - Six Sikkim Herald editions were acquired, page-screened, and font-audited
without creating labels. The source contract selected two State
Portal-hosted editions per script: Kirat Rai 2024-09-26/2025-01-13 (109
days), Lepcha 2024-07-26/2024-10-04 (70 days), and Sunuwar/Jenticha
2024-04-12/2025-01-26 (289 days). All six full PDF hashes and all 26 page
renders were frozen after an
omacrun. Reviewer 1 inspected every page at original render resolution: 25 pages have no explicit third-party marker observed, while both complete press-photo rectangles on Lepcha 2024-10-04 page 1 are bounded for exclusion because incidental storefront signage is visible. Both Lepcha PDFs have a structurally and visually blank page 5. This is a one-reviewer observation under the State Portal's custom permission, not a public-domain or government-authorship finding; reviewer 2 and adjudication remain null. - The revised live-layout converter audit reaches 100% aggregate coverage on
both Kirat Rai and both Sunuwar issues. Kirat raw body-unit totals are
13,101 and 7,920. Exact comparison to the
kirat rai font newprogram at xref 536 of official Unicode proposal L2/22-043R resolves the newly observed HeraldTthrough canonicalZto U+16D6C; the isolated extractedChas an empty outline and normalizes to space, and original-resolution review establishes that the custom glyph under:is a double quotation mark. The pair has 91 observed semantic source-code matches, zero mismatches, and zero unresolved units. Sunuwar raw totals are 9,697 and 14,676. Both font subsets match the exactkirat1program in the citedsunuwar_01348c63.pdfderivation source on every observed script code (76 summed matches, zero mismatches); backslash is an empty glyph, and the one U+201D extraction value visibly prints as a colon. The official proposal PDF hash is88b1d4dfedd8b36a687533f3fa1ef0a97c42965fd595b28678ca8d7d17cb8f17and its embedded reference font hash isbab945f8a5fe51f9401dc2a55a54ac99f3ba2b5071ac72324a05e7ff82cb4149; the Sunuwar reference PDF/font hashes areb3c001ff428de3b4e539c88d6d93566ae969a46b9ba947fe680fc1b68eb10456and484550543232f587fab3a5b3e900c922845c5ed44ef7bdd493cace4a5b6bce78, tied to MIT-licensednepal-ttf2utf0.2.0 ata158a8cbdbe230df6849afd9c9736bf31cb9c24b. The two 2024 Lepcha bodies are raster-composed, with 640 and 2,212 image placements, so converter coverage is N/A rather than 0%. - The joined bundle passes adversarial metadata audit but remains
NOT_ADMITTED. Mutation tests reject any label-write claim, injected text/label field, unreconciled counter, font mismatch, or invented second reviewer. The frozen audit joins the exact source review, 26-page reviewer-1 record, revised converter artifact, and reference config; it records four live editions at 100%, two raster N/A editions, zero labels, zero native reviewers, no GPU/cloud/training/scoring, and no stored source or converted text. Exact and substring leakage are explicitly N/A with zero rows, not passed. Before any line can exist, the route still needs independent provenance reviewer 2/adjudication and native-script pixel/label review; only then may it create an edition-isolated train/eval split, run exact plus substring leakage audit, train, and compute codepoint CER. Artifacts:outputs/sikkim-herald-six-edition-source-review-2026-07-15-r11/,outputs/sikkim-herald-six-edition-manual-review-r1-2026-07-15-r11/,outputs/sikkim-herald-six-edition-converter-layout-audit-2026-07-15-r11b/, andoutputs/sikkim-herald-r11b-adversarial-bundle-audit-2026-07-15-r11/. - A blinded Sikkim Herald provenance-review handoff is frozen, but no second
review is claimed. The reviewer-2 packet links all 26 exact full-page
renders at original resolution without copying images or exposing reviewer-1
decisions. Its byte-bound worksheet requires a distinct reviewer 2 who did
not inspect reviewer-1 decisions before completion, a third distinct
adjudicator, exact six-edition/26-page coverage, and one resolution for every
disagreement and none for an agreement. Mutation tests reject prefilled
decisions, reused identities, missing/duplicate pages, out-of-page exclusion
geometry, label/prediction payload keys, and incomplete adjudication. The
independently frozen packet audit verifies all 26 HTML links and hashes and
reports
PASS_READY_NOT_ADMITTED; the real worksheet remains blank, so reviewer 2, adjudication, native line review, rows, leakage, a model, and CER remain absent. Artifacts:outputs/sikkim-herald-six-edition-provenance-review-r2-packet-2026-07-15-r11/andoutputs/sikkim-herald-six-edition-provenance-review-r2-packet-audit-2026-07-15-r11/. - The recovered Newar-Devanagari whitespace rewrite is mechanically exact,
but the underlying pack fails admission. A 64 GiB
omacrun independently reproduced all 1,820/565 train, 634/353 dev, and 427/142 test removals for the two narrowly defined artifact classes, matching all 6,763 normalized labels and leaving zero targeted patterns. All referenced PNGs decode and contain at least 16 dark pixels; there are 6,757 unique image hashes, four intra-train duplicate groups (six rows beyond the first), zero conflicting labels within those groups, and zero cross-split image hashes. The honest split audit fails: train versus dev+test has 18 exact rows and 575 rows with a shared 16+-codepoint substring (including exact), while dev versus test has zero exact but 136 substring rows. The recovered labels also lack source URL, PDF hash, page, bbox, date, and edition fields, and Gorkhapatra rights remainRISK. A deterministic non-native geometry review saw ink in all seven risk-selected samples but obvious neighboring-line fragments in two of the three widest crops; it makes no population-rate or label-faithfulness claim. No filtering, training, prediction, score, or CER was run in this audit round. Artifacts:outputs/newar-devanagari-normalized-claim-boundary-audit-2026-07-15-r12/andoutputs/newar-devanagari-normalized-visual-sample-review-2026-07-15-r12/.
Gurung Khema official-print reopening (2026-07-15 r13)
- The former “synthetic-by-nature” source conclusion is refuted. Two
official State Portal Sikkim Herald Gurung Editions contain running Gurung
Khema print across eight pages and are 52 days apart. The 2022-03-01 PDF is
SHA-256
e667bbd2eb4f6769209013fe569cdc37d3e1aff40a9f0121301b43a03afe5d14; the 2022-04-22 PDF is2db16c7b7abe678e02103037397c7a35e21fb2fd34d4e4413d1979d1cf7f7c93. The pages contain 21,106 text-layer characters. April page 4 prints issue 4 while pages 1–3 print issue 5; the frozen source record preserves this anomaly. Reviewer 1 inspected every original-resolution page and observed no explicit third-party marker, but that is not a government-authorship or license-cleanliness finding. - The legacy font has a fail-closed Unicode 16 conversion path. Unicode
proposal L2/22-096 states that the republished 2022 Herald uses the same
legacy KHEMA font as the proposal and supplies the legacy key semantics. Both
Herald PDFs use U+F0xx PUA re-encodings of those legacy bytes. Their 56 shared
cmap codes have exact decomposed outlines and advances. For each edition, all
53 compared observed mapped glyphs rank first against the expected proposal
glyph; the minimum observed binary-mask IoU is 0.490751. The converter handles
direct keys, the Herald C/N/X printed-shape aliases, O/V combining-mark
unifications, adjacent pre-base T/E reordering into Unicode logical order,
and paired
DDas double danda. It rejects ambiguous input rather than guessing. - Mechanical eligibility is 749 of 759 candidate lines, not a dataset. The March edition has 382 eligible and 6 rejected lines; April has 367 eligible and 4 rejected. Across both editions, the 10 rejections are four lines with no Gurung Khema character, five non-adjacent pre-base-medial sequences, and one isolated legacy danda unit. The audit serializes no source or converted strings and writes no crops or labels.
- The route remains
NOT_ADMITTED. The separately frozen reviewer-2 packet links all eight hash-bound source renders, exposes no reviewer-1 decisions, and contains a byte-identical blank worksheet. Its adversarial audit reportsPASS_READY_NOT_ADMITTED. An independent reviewer 2 and distinct adjudicator, then native pixel-label review, crop construction, an edition/manuscript-isolated split, exact plus substring leakage audit, and codepoint-CER scoring are still required. There are currently zero labels, predictions, scores, training rows, or CER values. The joined r13 adversarial audit also pins every source/font/review/packet hash, rejects mutated counts, glyph ranks, blinding, payloads, and admission, and exercises the real freezer to prove that the blank worksheet fails before writing an output directory. Artifacts:outputs/sikkim-herald-gurung-khema-two-edition-source-review-2026-07-15-r13/,outputs/sikkim-herald-gurung-khema-font-semantic-audit-2026-07-15-r13/,outputs/sikkim-herald-gurung-khema-manual-review-r1-2026-07-15-r13/,outputs/sikkim-herald-gurung-khema-provenance-review-r2-packet-2026-07-15-r13/, andoutputs/sikkim-herald-gurung-khema-provenance-review-r2-packet-audit-2026-07-15-r13/, plus the joined verdict atoutputs/sikkim-herald-gurung-khema-r13-adversarial-bundle-audit-2026-07-15-r13/.
Corrected claim numbers (2026-07-16, fixes executed on omac)
- Ol Chiki eval pack v3 (blank-guard builder, mechanism: white-fill invisible text in the source PDF): 772 clean lines (123 blank/near-blank dropped — the <16-dark-px guard caught 56 more than the census's fully blank 67). olchiki-v2 scores: public headline (CC0 issue) cp CER 0.0961 / exact 0.193 (n=358); September subset 0.0704 (n=414); combined 0.0826 / 0.2034 (n=772, internal until September license confirmed). Supersedes all prior Ol Chiki numbers.
- Newar-Devanagari whitespace-normalized rescore (newar-ft-v1, frozen 1,139-line test, scored on omac): old artifact-bearing 0.0466; raw predictions vs normalized labels 0.0501 (the model reproduces training artifacts); both-sides-normalized cp CER 0.0441 / exact 0.2888 — the claim-comparable number (labels at outputs/newar-pack-v1-normalized-2026-07-15, utility src/ocrtech/devanagari_whitespace.py).
Tirhuta real substrate delivered (2026-07-17)
First-ever real Tirhuta training text: outputs/videha-tirhuta-silver-2026-07-17/
— 4,148 clean converted lines (labels.txt) from Videha issues (CC BY-SA
4.0, per-issue provenance), zero-unmapped-only policy. Extension-byte drop is the
dominant loss (e.g. first issue: 1,235 extension-dropped of 7,622 examined)
— building the conjunct extension-glyph map would multiply yield; per-issue
numbers in the pack summary.json. Also staged and VERIFIED: Ol Chiki v3 fine-tune payload
(outputs/olchiki-v3-ft-payload-2026-07-17/, 5,330 entries) — train is 242
real Public-Domain-Mark-issue lines + 5,000 synth replay, September 414
eval-only with a zero-intersection leakage audit on filename/line-id/label;
READY-PENDING-APPROVAL in the GPU dossier (see
outputs/overnight-consolidation-2026-07-16/gpu-batch-proposal.md, candidate 3
update). Operational note: the Codex quota briefly hit its limit on 2026-07-16
(reset message named 2026-07-22) but lifted the same night — fan-out resumed
with the Tirhuta extension-map and Lepcha zero-yield lanes.
Lepcha zero-yield root cause (2026-07-17)
The Sikkim Herald Lepcha lane's 0 lines (vs Sunuwar 9,743 / Kirat Rai 3,699)
is a data reality, not a bug: in all 18 editions the Lepcha article body is
rasterized as image XObjects — there is no live Lepcha text layer (live text
is English masthead/punctuation in Times New Roman / anonymized CID fonts;
zero U+1C00-1C4F codepoints in sampled pages). The converter was never
invoked. Full evidence: outputs/lepcha-zero-yield-diagnosis-2026-07-17/REPORT.md.
Consequence: Lepcha real-line substrate requires a raster OCR/transcription
lane (or stays synth-first per the headline plan); the builder should record
raster_body_requires_ocr instead of wrong_font for such editions.
Tirhuta extension map + v2 pack: yield 2.6x (2026-07-17)
Codex built the Videha Tirhuta extension-glyph map (149 new mappings, each
grounded in font glyph semantics + Pandey L2/11-175R; 2,131 converter tests
pass) — branch agent/tirhuta-extension-map in nepal-ttf2utf, commit
a5ba521. Rebuilt the pack on omac with the new converter:
outputs/videha-tirhuta-silver-v2-2026-07-17/ — 10,696 clean lines
(vs 4,148 in v1; 45,157 examined; extension-byte drops now 1, down from
thousands). Pack verified (archive files == labels == manifest == 10,696),
backed up to HF ampixa/ocr-tech-artifacts. Builder bug found during
finalize: str.splitlines() splits JSON-legal raw U+0085 in provenance
strings — fixed to split("\n") (patched builder stored in the pack dir).
Remaining converter work: merge the branch to main after review.
Three FT payloads staged (2026-07-17)
Codex assembled per-script fine-tune payloads on the Ol Chiki v3 discipline (issue/page-boundary dev splits, four-way zero-leakage audits, sha-pinned manifests, training_launched=false):
- Tirhuta READY: outputs/tirhuta-v2-ft-payload-2026-07-17/ — 9,765 real train / 844 dev (whole issue videha_01_02_08 held out), 10,639 entries, sha 55c18998...13e4. CC BY-SA 4.0 with attribution recorded.
- Sunuwar BLOCKED-PENDING-LEGAL: outputs/sunuwar-v1-ft-payload-2026-07-17/ (8,422/514) — Sikkim Herald State-Portal permission + India §17(d) question unresolved; assembled but not trainable yet.
- Kirat Rai BLOCKED-PENDING-LEGAL: outputs/kiratrai-v1-ft-payload-2026-07-17/ (2,865/193) — same gate.
Verified: no synth replay pools or warm-start checkpoints exist for these three scripts anywhere in outputs/ or HF artifacts (the native-script wave built substrate, not models; only Ol Chiki trained). Consequence: the proven synth-pretrain-then-real-FT recipe needs a Tirhuta synth pack before the Tirhuta payload trains warm; alternatively from-scratch real-only must be an explicit dossier decision.
Tirhuta synth-v1 pretrain pack rendered (2026-07-19)
The missing pretrain stage for Tirhuta is now built:
outputs/tirhuta-synth-v1-prep-2026-07-17/tirhuta-synth-v1-rendered.tgz
(sha 2bf91b49...1327, 48,960 PNGs = 8,160 texts x 6 degradation profiles,
7,997 train / 163 eval texts, zero normalized collisions with the real FT
dev set, Noto Sans Tirhuta OFL only, RAQM shaping verified, dark-pixel
guard min 102). Rendered on omac; archive re-extracted and hash-verified;
backed up to HF. Engineering notes: macOS Pillow lacks RAQM — fixed via
Xcode CLT + brew libraqm + DYLD_FALLBACK_LIBRARY_PATH exported INSIDE the
script (SIP strips DYLD_* across /bin/zsh exec); audit had a /tmp vs
/private/tmp resolve() mismatch, patched in the pack's audit script.
Tirhuta training chain is now fully staged: synth-v1 pretrain ->
tirhuta-v2-ft real payload (dossier candidate 5); GPU run still
human-gated.
Next Work
Post-recovery next (2026-07-05): the spike verdict keeps the Paddle det + Rows-LTR + y/conf70 chain as the backbone. Phase-2 pivot, measured 2026-07-05: decomposing the 0.4134 filtered-baseline page CER on the 16-page pack showed the standing "region selection + furniture" assumption is wrong — deleting every unmatched prediction (perfect furniture/region filtering) changes CER by ~0 (0.4152), while reordering the matched lines into reference order alone drops it to 0.0378 (10.9×). Reading order is 91% of page CER. Geometry heuristics cannot recover it (Columns LTR 0.799, PP-Structure-region banding 0.702, PP-Structure native order 0.797) because the reference order is the PDF's article-box emission order, not a row/column pattern. Three follow-up experiments (all 2026-07-05, artifacts under
outputs/reading-order-model-v1-2026-07-05/andoutputs/line-merge-v1-2026-07-05/) then closed the question:- Learned reading-order model v1: refuted. LightGBM pairwise
comparator on the silver
span_bboxorder supervision regressed to 0.799 — the Devanagari packs preserve PDF article-box emission order (not row-major) while the Limbu eval reference is a strict y-sort; wrong supervision for the target. Leakage-clean; kept as a negative result. - Detector-fragmentation hypothesis: refuted. Det boxes ≈ 1:1 with reference lines (mean best-IoU 0.68; the widest lines average 1.05 fragments). Tuned merge (gap 0.6×h) merges ~3 boxes/page, CER flat. Naive merge (gap 1.0×h) bridges the 40 px column gutters and collapses det F1 to 0.71 — gutters are narrower than line height on these pages.
- Root cause found: the reference linearization is our own artifact.
build_limbu_gorkhapatra_all_corpus.py:209sorts spans strict y-then-x across a 6-column layout; det-vs-PDF y-jitter of a few px flips column interleaving constantly (pair accuracy 0.998, page CER still 0.30–0.41). Pure-y prediction sort gives a free win (0.4134 → 0.3006), and one shared deterministic column-major linearization applied to both reference and prediction yields page cp CER 0.0451 — at the content floor (~0.054: recognition 0.026 + 211 missed / 87 extra lines). Page reconstruction on this pack is solved up to linearization definition; the honest next metric is page-eval v2 with column-major linearization on both sides (a documented metric change — not comparable to the old 0.41-scale numbers), and the production assembly should emit column-major order. Remaining real levers after v2: det recall/precision (211/87), recognizer (0.026), and native review for claim-grade.
Page-eval v2 + Column Major assembly shipped 2026-07-05: the shared linearization lives in
src/ocrtech/linearization.py(tests intests/test_linearization.py);tools/apply_page_reading_order_strategy.pygained mode"Column Major";tools/run_page_ocr_eval.pygained--page-linearization {native,column-major}(recorded inpage-eval-run.json, per-sample fallback to native recorded when a reference lacks line geometry). Validated on the 16-page pack: Column-Major assembly + v2 scoring gives page cp CER 0.043 / grapheme 0.067 (outputs/limbu-column-major-2026-07-05-score-v2/); the native path is regression-identical (0.413/0.437). v1-scale and v2-scale numbers must never be compared directly.Newar page-eval v2 (second-language proof, 2026-07-05/06): a 19-page Newar page pack built from the rebuilt harvest (
tools/build_newar_gorkhapatra_silver_page_eval.py,outputs/newar-page-eval-v2-2026-07-05/pack/) and scored through the identical non-oracle chain (Paddle det → recognizer → Column Major → y/conf70 → page-eval): newar-ft-v1 native 0.778 → v2 0.199 page cp CER (grapheme 0.279; det F1 0.912; matched-line cp CER 0.0905) — the column-major linearization story replicates exactly on the second language; the Newar residual is recognition + det-crop domain gap, not reconstruction. Two follow-ups surfaced and both were fixed 2026-07-06: (a) the deployedlimbu-devanagari-ppocrv5export was unusable as a general Devanagari recognizer (409-byte dict missing anusvara, ii-matra, ṇa…; emitted garbage — its page numbers were a formality, not a model comparison); the recovered deva-v2 training checkpoint was re-exported with its correct 144-chardevanagari-v2.txtand verified to reproduce the recorded pre-crash Newar baseline exactly (cp CER 0.062229 / grapheme 0.13120 / exact 0.158 on the recovered 1,139-line frozen test); the Space now serves it asmodels/deva-v2(export also atoutputs/deva-v2-reexport-2026-07-06/and on the HF dataset). (b) The Newar pack builder's digest mismatch was a computation-path divergence from the eval harness's recomputation — builder fixed (9 tests green), rebuilt pack atoutputs/newar-page-eval-v2-2026-07-06-digestfix/scores with zero digest warnings and identical CER (0.19918).Phase 3 done 2026-07-05: config-driven
tools/run_book_ocr.pylanded (det → script-route → per-script recognizers fromconfigs/nepal-language-pack-pipeline-v1.json→ Column Major default → filter → Unicode/MD export + review queue, resumable, preflight validation; smoke on 2 rebuilt Limbu pages: 606 lines kept / 25 queued). The rebuilt Limbu pack is 2× the lost one (16 pages / 4,872 labels).ft-v2 retrain result (2026-07-06): new champion candidate. 40 epochs on the L4 (~8.5 GPU-h), 46,753 rows (25,065 synth + 2,711 real ×8 replay), PP-OCRv5 mobile generic base, selection on real dev (best epoch 39). On the frozen 902-line test (dates 2082-03-24 / 2082-08-12 / 2082-12-25, by-date disjoint, leakage-gated), scored through the identical local CPU path as the deployed champion: ft-v2 cp CER 0.0047 / grapheme 0.0098 / exact-line 0.859 vs champion 0.0138 / 0.0284 / 0.637 — ~3× lower CER, +22 pts exact-line. The gap is if anything understated: the old champion's training pack drew on the old harvest and may have seen these dates; ft-v2's test isolation is proven. We also now own a warm-startable training checkpoint again (
outputs/limbu-ft-v2-run-2026-07-06/checkpoints/best_accuracy.*; export at.../export/). Silver labels, diagnostic, not claim-grade. Acceptance gate PASSED (2026-07-06): ft-v2 through the full non-oracle 16-page chain (same detector, Column Major, y/conf70 filter, page-eval v2): page cp CER 0.0272 / grapheme 0.0347 vs champion 0.043 / 0.067; matched-line cp CER 0.0090 vs 0.0266 (−66%); det F1 unchanged (0.9650 vs 0.9657, same detector). 3,725 of 4,699 matched lines exact. Gate artifacts:outputs/limbu-ftv2-page-gate-2026-07-06/. Real Gorkhapatra newspaper pages now read at ~2.7% codepoint error in human-readable column-major order, silver-diagnostic boundary unchanged. Remaining step: package ft-v2 (model card, hashes) and swap the Space bundle — a production-facing change, pending user confirmation.- Learned reading-order model v1: refuted. LightGBM pairwise
comparator on the silver
Finish the 2026-07-04 full harvest and inventoryDone 2026-07-04: 394 language-page hits, 50 languages, inventory manifests atoutputs/language-pack-inventory-full-2026-07-04/.Per-language silver packs via the existingDone 2026-07-04 (nayanepalconvertertools/build_language_pack_silver_line_packs.py, tests green): the batch-1 run produced 116,287 kept silver lines across 50 languages, zero errors, with per-line provenance (PDF hash, page, bbox, raw legacy bytes, converted text, image hash) atoutputs/language-pack-silver-lines-2026-07-04/; spot-checked conversions read as clean, language-plausible Devanagari. The batch-2 run over the older 589 page hits kept 183,324 more lines (49 languages, zero errors) atoutputs/language-pack-silver-lines-batch2-2026-07-04/— 299,611 per-language-attributed silver lines total across the two batches (~2.4 GB of crops). Run the tool withuv run --python 3.11 --with pymupdf --with npttf2utf—nepal_ttf2utfneeds thenpttf2utfbase-map package at runtime, and without it every span fails (cleanly, with per-span error records, but kept=0). Honest caveat: lines are page-attributed, not language-verified — a language page also carries Nepali furniture/boxes, and pages whose body is not nayanepal Devanagari (Urdu, the Tibetan-script pages) yield mostly furniture lines. A language-ID/lexicon filter before per-language recognizer training is the follow-up gate.Newar track (second focus language): add the Newar registry entry (Devanagari + Prachalit rows); build the Devanagari-written Newar recognizer eval/train split from the 6,800 harvested silver lines (by-date split, frozen test); keep native Prachalit as the separate longer arc — real printed-register Nepal Lipi harvesting, not more transliteration, is the lever.
Classify Tibetan-script pages (Hyolmo/Mugal/Bhote) and decide converter vs native Unicode.
Generalize the pipeline wiringDone 2026-07-04: all three Limbu-locked spots generalized (shared script table, filter keys,--required-scriptpreflight), tests green;configs/nepal-language-pack-pipeline-v1.jsonholds the per-language wiring.Target-region/furniture filtering stage for Paddle page/book OCR, scored against the current baseline on the 8-page Limbu pack and at least one Devanagari-language pack (Newar is the natural candidate given the focus order).
Only after page reconstruction improves, export high-quality real/silver line crops back into recognizer training, per language.
Month-batch the archive back-harvest toward the full ~1,750-article depth, storing the mirror on the remote 2 TB volume.
Mistakes To Avoid
Do not use Naya Nepal blog images as training/eval evidence when the PDF exists. Do
not use wide epaper date windows — the endpoint silently caps at 30 PDFs per query;
weekly windows only. Do not treat unnormalized Gorkhapatra labels as distinct
languages (थारु vs थारू). Do not treat the nayanepal text layer as Unicode — it is
legacy Preeti-family and needs the converter before any silver label is real. Do not
report Paddle norm_edit_dis as CER; real CER comes from tools/eval_metrics.py
micro CER over canonical predictions. Do not treat synthetic data as claim evidence.
Do not call Gurung Khema synthetic-by-nature now that official Herald print is
frozen, and do not promote its 749 mechanically eligible candidates to rows before
independent provenance, native label, crop, split, and leakage review. Do not promote the
targeted+Unicode checkpoint or v3 dictionary warm-starts. Do not spend long blocks on
generic eval-harness work unless it directly improves the book/page OCR loop. Do not
interpret a syntactically valid approval/auditor identity as cryptographic proof;
retain independent custody or signed runner evidence for a production claim. Do not
extend the State Portal's Sikkim Herald permission to the separate IPR PDF origin;
use ipr.sikkim.gov.in as discovery-only unless a file is mirrored or separately
permitted. Do not frame video-to-page work as OCR scope; reduce video to stills
first. Do not present
page OCR as solved: matched-line recognition is strong, but full-page CER is still
dominated by reconstruction and region selection.