# Devanagari OCR resources inventory (2026-09-24) Scope: everything usable for a Nepali/Hindi-family Devanagari line recognizer that could be found on our machines, our Hugging Face accounts, and public sources, checked on disk or live on 2026-09-24. Counts are read from the artifacts, not recalled. Use decisions follow `docs/DATA_LICENSING.md`; nothing here is claim-grade ground truth unless stated. ## Our machines | Where | Artifact | What it is | Status / use | |---|---|---|---| | TRANSCEND (`ocr-tech/`) | `data/gorkhapatra-limbu-editions/epaper-*.pdf` | 24 Gorkhapatra editions (2081-07-12 .. 2083-02-12), 493 MB, `nayanepal` (Preeti-family) text layer | Primary real source. Rebuilt into `chunks-v2` (below). Internal training only (owner determination 2026-07-22). | | TRANSCEND | `data/gorkhapatra-devanagari-real/` | 79,364 converted line labels from those editions; only 40,359 of the line images survived the July crash (`docs/RECOVERY_LEDGER.md`) | Superseded for training by `chunks-v2`; surviving edition 2082-03-24 supplies the long-line test E1b. | | TRANSCEND | `outputs/newar-devanagari-silver-pack-v1-recovered/` + `outputs/newar-pack-v1-normalized-2026-07-15/` | Newar-language Devanagari lines 4,096 / 1,528 / 1,139 | `NOT_ADMITTED` for claims (leakage, provenance). Test split used only as a secondary diagnostic (E2), with train chunks filtered against it. | | TRANSCEND | `outputs/language-pack-silver-lines-rebuild-b{1,2}-2026-07-05.tgz` | 122,882 + 184,089 page-attributed silver lines, ~50 languages (Naya Nepal pages) | Not used yet: line-level language ID and furniture filtering still required. | | TRANSCEND | `outputs/gorkhapatra-backfill-2024-03 .. 2025-07-*` | 515,586 page-attributed silver lines (monthly archives) | Not used yet (same filtering requirement); largest untapped real-print pool. | | TRANSCEND | `outputs/deva-v2-reexport-2026-07-06/`, `spaces/limbu-ocr/models/deva-v2/` | Our Devanagari recognizer; 143-entry dictionary with only 28 Latin letters and ASCII digits 0,1,2,3,7 | Replaced on the k2 demo by the official Paddle model on 2026-09-24. | | TRANSCEND | `data/fonts/all/devanagari/` | 289 font files | 48 families verified OFL (Google Fonts `ofl/` or embedded OFL name records, incl. Annapurna SIL Nepal, Aarav). Excluded: Kalimati, Gargi, Yashomudra, Sanskrit2003 (no licence), Kohinoor (all rights reserved), Siddhanta (CC BY-NC-ND), Chandas/Uttara/Samyak (GPL), NithyaRanjanaDU (Ranjana glyphs). | | k2 (`~/ocr-tech`, not git) | `data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl`, `data/splits-mixed-v1/` | 2026-06 Nepali+English recognizer work (18,765 gauravgiri rows; mixed-v1 1,017 train / 180 eval) | Source licence unrecorded; kept out of training. | | k2 | `data/manifests-english-test/` | 1,000 NVIDIA English line crops from the earlier run | Superseded by `nvidia-en-lines` (below). | | k2 | `runs/paddle-recognizer-mixed-v2-bundle/`, `runs/hf-recognizer-*` (~10 GB) | Earlier Nepali/English recognizer experiments | Historical; not reused. | | k2 | `~/.paddlex/official_models/` | `devanagari_PP-OCRv5_mobile_rec`, `PaddleOCR-VL-1.6`, `PP-DocLayoutV3`, `PP-OCRv5_server_det`, `PP-OCRv6_medium_det`, `UVDoc`, orientation classifiers | Official Devanagari model now serves `ampixa.thapa-ashish.com.np`. | | cdjk | `~/deva-ft-build/` | Build workspace: edition copies, OFL font subset, `chunks-v2`, `synth-v1` | Canonical build location for the 2026-09-24 fine-tune data. | | mbp2 | `~/github/llm/ocr-tech` | Stale 2026-07-04 checkout (712 MB `data/`) | Nothing newer than TRANSCEND. | | omac | — | Offline since ~2026-08-01 | Unreachable. | ## Our Hugging Face repos | Repo | Relevant content | Use | |---|---|---| | `ampixa/ocr-tech-artifacts` (private) | `deva-v2-reexport`, Newar packs, `aws-newar-recovery`, detector datasets | Backups of the items above. | | `ampixa/nepal-mt-text` (private) | Nepali (`npi`) sides of MT corpora; licences mostly unrecorded or Bible text | Weak rendering source; not used. | ## Official models | Model | Devanagari coverage | Notes | |---|---|---| | `devanagari_PP-OCRv5_mobile_rec` (Paddle) | 568-entry dictionary: 128 Devanagari, all 52 Latin letters, 0-9 | Training weights (`*_pretrained.pdparams`, 70.5 MB) and config are published; safetensors (`PaddlePaddle/devanagari_PP-OCRv5_mobile_rec_safetensors`) and ONNX repos exist. | | PP-OCRv6 | 0 Devanagari characters | Not usable for Nepali. | | PaddleOCR-VL-1.6 (0.9B, Apache-2.0) | Lists Hindi among 109 languages | Candidate offline teacher; Nepali quality unverified. | | `transformers` 5.17 `pp_ocrv5_mobile_rec` | PyTorch port of the same network | Used for MPS fine-tuning on k2. | ## Public datasets (checked 2026-09-24) | Dataset | Content | Licence | Decision | |---|---|---|---| | `nvidia/OCR-Synthetic-Multilingual-v1` | Synthetic pages; en/ja/ko/ru/zh only (no Devanagari) | CC BY 4.0 | English replay: 6,802 horizontal train lines + 419 page-disjoint test lines from `en/validation/validation_000.h5`. | | `sarvamai/indic-ocr-bench` | 6,909 block-level samples, 23 languages, reviewed twice | Apache-2.0 | Future independent eval (needs page/block pipeline). | | `himalaya-ai/nepali-deva-ocr-eval` | 500 rows sampled from `gauravgiri/nepali-ocr-dataset`; the images are photographed **handwriting**, some labels mismatch | none recorded | Eval-only handwriting probe (E5); never trained on. | | `himalaya-ai/devanagari_ocr_dataset` | Compilation of 7 public sets, ~58 GB | Apache-2.0 tag over mixed upstream licences | Parked (size; upstream licences). | | `iamkushagratomar/devanagari-ocr-synthetic-80k` | Hindi synthetic lines, 8 fonts | MIT | Optional extra font variety; not used in v1. | | `Om-7387/Marathi-OCR`, `meharuhanzz/OCR-Bench1000-{Hindi,Marathi}` | Small Marathi/Hindi sets | Apache-2.0 / MIT | Candidate evals. | | IIIT Mozhi (`darknight054/indic-mozhi-ocr` mirror) | ~80k printed lines per language incl. Hindi, Marathi | No licence stated (checked CVIT page) | Excluded until CVIT confirms terms. | | IndicSTR12 | Scene-text words, 12 languages | Disputed (reported CC BY-NC-ND) | Excluded; wrong domain. | | `thegauravgiri/nepali-ocr-dataset`, `vishwam-101/devanagari-ocr-dataset`, `Titung/nepali-books-ocr-dataset-full` | Nepali/Devanagari line sets | none recorded | Excluded from training. | ## Built for the 2026-09-24 fine-tune | Set | Size | Built by | |---|---|---| | `chunks-v2` train / dev / test | 173,167 / 24,908 / 33,375 real print chunks (<= 25 codepoints), edition-disjoint (test 2082-03-24, 2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11) | `tools/build_gorkhapatra_devanagari_chunks.py` (pre-fix labels: carry the क्र->व्र bug) | | `chunks-v4` train / dev / test (used for the deployed v3) | 125,017 / 15,570 / 21,821 chunks (<= 60 codepoints), same editions, labels converted with the `j|m` -> `s|` क्र fix (3,425 spans corrected, 2,801 previously rejected spans recovered) | same tool with `RAW_FIXES` | | `synth-v1` train | citizenship_ne 30,000; office_ne 12,000; citizenship_en 10,000; realtext 30,000 (44 OFL families) | `tools/render_devanagari_form_synth.py` | | `synth-v1` dev / test | 800 dev (train fonts, separate seed); 1,800 test in 4 held-out families (Sura, Khula, Dekko, Sarala) | same | ## Handwriting (checked 2026-09-24) v3 is worse than the official model on handwriting (it was fine-tuned on print only), so a handwriting fine-tune (v4) is staged. Data was downloaded on cdjk (`~/hw-data/`, recipes in `tools/deva_rec_torch/recipes/cdjk/`) and IIIT directly on k2; sets are staged on k2 under `~/ocr-deva-ft/data/hw-train/` and `hw-eval/`. | Dataset | Content | Licence | Use | |---|---|---|---| | IIIT-HW-Dev (IIIT-INDIC-HW-WORDS Devanagari part; HF `c3rl/IIIT-INDIC-HW-WORDS-Hindi`, also inside Kaggle `himanshukumarrajak/indic-data`) | 95,430 handwritten Hindi words | research and educational use only (CVIT) | H2: 69,853 train / 2,000 dev / 2,000 test. **Any model trained on it is a research model** until CVIT grants permission. | | DHCD, UCI id 389 | 92k isolated 32x32 characters, incl. 20k digits ०-९ | CC BY 4.0 | H4: 15,000 composed digit strings (dates, certificate/ward/phone numbers, amounts) from DHCD Train digits; 800 test strings from DHCD Test digits | | Kaggle `sweekardahal/handwritten-labeled-dataset-for-text-recognition` | 4,357 Nepali handwritten word crops | CC0 | H1: 3,187 train / 300 dev / 871 test (random word split) | | Kaggle `sweekardahal/nepali-handwritten-images-for-text-detection` (NHD) | Nepali handwritten page photos with word boxes, no text | CC0 | Downloaded; unused (future handwriting detector) | | Kaggle `rushdaismailaslami/handwritten-devanagari-character-and-word-dataset` (IndicScan) | 835 labelled Hindi handwritten words | CC BY 4.0 | H3: test only | | Kaggle re-uploads of IIIT data under other tags (`daddyshomeee` MIT, `sabarinathan` CC BY-NC, `anshmaurya420`) | IIIT data | tags on re-uploads | Treated as IIIT: research only | Baseline (served path, codepoint CER, `tools/deva_rec_torch/results/hw_baseline.json`): H1 official 40.8% vs v3 68.9%; H2 49.4% vs 60.5%; H3 25.2% vs 45.8%. ## Page-level check on real letters (2026-09-24) `tools/deva_rec_torch/page_probe/`: two public ocr.gov.np notices, 43 hand-transcribed printed lines (eval-only smoke test). The worker's axis-aligned line crops give 21.4% codepoint CER; rotated min-area crops with 6 px padding give 2.9%. PP-DocLayoutV3 found the QR codes and emblems but labelled neither stamp as `seal` and has no signature or handwriting class. Details in `RESEARCH.md`. ## Technical findings that constrain the approach - Paddle CPU training on macOS arm64 (k2) runs at ~2.2 images/s: the PPLCNetV3 backbone forward alone takes 5.8 s per 32-image batch, `set_num_threads` has no effect, and the wheel has no MKLDNN. PyTorch MPS on the same machine trains the same network at ~70 img/s. - PaddleX recognition preprocessing pads to `int(48 * max(320/48, w/h))` and caps the aspect width at it (floor for wide crops). A 1-px width difference flips low-confidence characters on ~40% of long lines (SVTR attention sees the padding), so the torch path mirrors it exactly (`tools/deva_rec_torch/common.py`, 98.8% line parity with PaddleX). - PaddleOCR's NRTR label encoder drops labels of `max_text_length - 1` or more characters (23 with the official 25), and `encode()` silently skips out-of-dictionary characters.