ocr-projects-archive / ocr-tech /docs /DEVANAGARI_RESOURCES_INVENTORY.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
10.4 kB

Devanagari OCR resources inventory (2026-09-24)

Scope: everything usable for a Nepali/Hindi-family Devanagari line recognizer that could be found on our machines, our Hugging Face accounts, and public sources, checked on disk or live on 2026-09-24. Counts are read from the artifacts, not recalled. Use decisions follow docs/DATA_LICENSING.md; nothing here is claim-grade ground truth unless stated.

Our machines

Where Artifact What it is Status / use
TRANSCEND (ocr-tech/) data/gorkhapatra-limbu-editions/epaper-*.pdf 24 Gorkhapatra editions (2081-07-12 .. 2083-02-12), 493 MB, nayanepal (Preeti-family) text layer Primary real source. Rebuilt into chunks-v2 (below). Internal training only (owner determination 2026-07-22).
TRANSCEND data/gorkhapatra-devanagari-real/ 79,364 converted line labels from those editions; only 40,359 of the line images survived the July crash (docs/RECOVERY_LEDGER.md) Superseded for training by chunks-v2; surviving edition 2082-03-24 supplies the long-line test E1b.
TRANSCEND outputs/newar-devanagari-silver-pack-v1-recovered/ + outputs/newar-pack-v1-normalized-2026-07-15/ Newar-language Devanagari lines 4,096 / 1,528 / 1,139 NOT_ADMITTED for claims (leakage, provenance). Test split used only as a secondary diagnostic (E2), with train chunks filtered against it.
TRANSCEND outputs/language-pack-silver-lines-rebuild-b{1,2}-2026-07-05.tgz 122,882 + 184,089 page-attributed silver lines, ~50 languages (Naya Nepal pages) Not used yet: line-level language ID and furniture filtering still required.
TRANSCEND outputs/gorkhapatra-backfill-2024-03 .. 2025-07-* 515,586 page-attributed silver lines (monthly archives) Not used yet (same filtering requirement); largest untapped real-print pool.
TRANSCEND outputs/deva-v2-reexport-2026-07-06/, spaces/limbu-ocr/models/deva-v2/ Our Devanagari recognizer; 143-entry dictionary with only 28 Latin letters and ASCII digits 0,1,2,3,7 Replaced on the k2 demo by the official Paddle model on 2026-09-24.
TRANSCEND data/fonts/all/devanagari/ 289 font files 48 families verified OFL (Google Fonts ofl/ or embedded OFL name records, incl. Annapurna SIL Nepal, Aarav). Excluded: Kalimati, Gargi, Yashomudra, Sanskrit2003 (no licence), Kohinoor (all rights reserved), Siddhanta (CC BY-NC-ND), Chandas/Uttara/Samyak (GPL), NithyaRanjanaDU (Ranjana glyphs).
k2 (~/ocr-tech, not git) data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl, data/splits-mixed-v1/ 2026-06 Nepali+English recognizer work (18,765 gauravgiri rows; mixed-v1 1,017 train / 180 eval) Source licence unrecorded; kept out of training.
k2 data/manifests-english-test/ 1,000 NVIDIA English line crops from the earlier run Superseded by nvidia-en-lines (below).
k2 runs/paddle-recognizer-mixed-v2-bundle/, runs/hf-recognizer-* (~10 GB) Earlier Nepali/English recognizer experiments Historical; not reused.
k2 ~/.paddlex/official_models/ devanagari_PP-OCRv5_mobile_rec, PaddleOCR-VL-1.6, PP-DocLayoutV3, PP-OCRv5_server_det, PP-OCRv6_medium_det, UVDoc, orientation classifiers Official Devanagari model now serves ampixa.thapa-ashish.com.np.
cdjk ~/deva-ft-build/ Build workspace: edition copies, OFL font subset, chunks-v2, synth-v1 Canonical build location for the 2026-09-24 fine-tune data.
mbp2 ~/github/llm/ocr-tech Stale 2026-07-04 checkout (712 MB data/) Nothing newer than TRANSCEND.
omac — Offline since ~2026-08-01 Unreachable.

Our Hugging Face repos

Repo Relevant content Use
ampixa/ocr-tech-artifacts (private) deva-v2-reexport, Newar packs, aws-newar-recovery, detector datasets Backups of the items above.
ampixa/nepal-mt-text (private) Nepali (npi) sides of MT corpora; licences mostly unrecorded or Bible text Weak rendering source; not used.

Official models

Model Devanagari coverage Notes
devanagari_PP-OCRv5_mobile_rec (Paddle) 568-entry dictionary: 128 Devanagari, all 52 Latin letters, 0-9 Training weights (*_pretrained.pdparams, 70.5 MB) and config are published; safetensors (PaddlePaddle/devanagari_PP-OCRv5_mobile_rec_safetensors) and ONNX repos exist.
PP-OCRv6 0 Devanagari characters Not usable for Nepali.
PaddleOCR-VL-1.6 (0.9B, Apache-2.0) Lists Hindi among 109 languages Candidate offline teacher; Nepali quality unverified.
transformers 5.17 pp_ocrv5_mobile_rec PyTorch port of the same network Used for MPS fine-tuning on k2.

Public datasets (checked 2026-09-24)

Dataset Content Licence Decision
nvidia/OCR-Synthetic-Multilingual-v1 Synthetic pages; en/ja/ko/ru/zh only (no Devanagari) CC BY 4.0 English replay: 6,802 horizontal train lines + 419 page-disjoint test lines from en/validation/validation_000.h5.
sarvamai/indic-ocr-bench 6,909 block-level samples, 23 languages, reviewed twice Apache-2.0 Future independent eval (needs page/block pipeline).
himalaya-ai/nepali-deva-ocr-eval 500 rows sampled from gauravgiri/nepali-ocr-dataset; the images are photographed handwriting, some labels mismatch none recorded Eval-only handwriting probe (E5); never trained on.
himalaya-ai/devanagari_ocr_dataset Compilation of 7 public sets, ~58 GB Apache-2.0 tag over mixed upstream licences Parked (size; upstream licences).
iamkushagratomar/devanagari-ocr-synthetic-80k Hindi synthetic lines, 8 fonts MIT Optional extra font variety; not used in v1.
Om-7387/Marathi-OCR, meharuhanzz/OCR-Bench1000-{Hindi,Marathi} Small Marathi/Hindi sets Apache-2.0 / MIT Candidate evals.
IIIT Mozhi (darknight054/indic-mozhi-ocr mirror) ~80k printed lines per language incl. Hindi, Marathi No licence stated (checked CVIT page) Excluded until CVIT confirms terms.
IndicSTR12 Scene-text words, 12 languages Disputed (reported CC BY-NC-ND) Excluded; wrong domain.
thegauravgiri/nepali-ocr-dataset, vishwam-101/devanagari-ocr-dataset, Titung/nepali-books-ocr-dataset-full Nepali/Devanagari line sets none recorded Excluded from training.

Built for the 2026-09-24 fine-tune

Set Size Built by
chunks-v2 train / dev / test 173,167 / 24,908 / 33,375 real print chunks (<= 25 codepoints), edition-disjoint (test 2082-03-24, 2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11) tools/build_gorkhapatra_devanagari_chunks.py (pre-fix labels: carry the क्र->व्र bug)
chunks-v4 train / dev / test (used for the deployed v3) 125,017 / 15,570 / 21,821 chunks (<= 60 codepoints), same editions, labels converted with the `j m->s
synth-v1 train citizenship_ne 30,000; office_ne 12,000; citizenship_en 10,000; realtext 30,000 (44 OFL families) tools/render_devanagari_form_synth.py
synth-v1 dev / test 800 dev (train fonts, separate seed); 1,800 test in 4 held-out families (Sura, Khula, Dekko, Sarala) same

Handwriting (checked 2026-09-24)

v3 is worse than the official model on handwriting (it was fine-tuned on print only), so a handwriting fine-tune (v4) is staged. Data was downloaded on cdjk (~/hw-data/, recipes in tools/deva_rec_torch/recipes/cdjk/) and IIIT directly on k2; sets are staged on k2 under ~/ocr-deva-ft/data/hw-train/ and hw-eval/.

Dataset Content Licence Use
IIIT-HW-Dev (IIIT-INDIC-HW-WORDS Devanagari part; HF c3rl/IIIT-INDIC-HW-WORDS-Hindi, also inside Kaggle himanshukumarrajak/indic-data) 95,430 handwritten Hindi words research and educational use only (CVIT) H2: 69,853 train / 2,000 dev / 2,000 test. Any model trained on it is a research model until CVIT grants permission.
DHCD, UCI id 389 92k isolated 32x32 characters, incl. 20k digits ०-९ CC BY 4.0 H4: 15,000 composed digit strings (dates, certificate/ward/phone numbers, amounts) from DHCD Train digits; 800 test strings from DHCD Test digits
Kaggle sweekardahal/handwritten-labeled-dataset-for-text-recognition 4,357 Nepali handwritten word crops CC0 H1: 3,187 train / 300 dev / 871 test (random word split)
Kaggle sweekardahal/nepali-handwritten-images-for-text-detection (NHD) Nepali handwritten page photos with word boxes, no text CC0 Downloaded; unused (future handwriting detector)
Kaggle rushdaismailaslami/handwritten-devanagari-character-and-word-dataset (IndicScan) 835 labelled Hindi handwritten words CC BY 4.0 H3: test only
Kaggle re-uploads of IIIT data under other tags (daddyshomeee MIT, sabarinathan CC BY-NC, anshmaurya420) IIIT data tags on re-uploads Treated as IIIT: research only

Baseline (served path, codepoint CER, tools/deva_rec_torch/results/hw_baseline.json): H1 official 40.8% vs v3 68.9%; H2 49.4% vs 60.5%; H3 25.2% vs 45.8%.

Page-level check on real letters (2026-09-24)

tools/deva_rec_torch/page_probe/: two public ocr.gov.np notices, 43 hand-transcribed printed lines (eval-only smoke test). The worker's axis-aligned line crops give 21.4% codepoint CER; rotated min-area crops with 6 px padding give 2.9%. PP-DocLayoutV3 found the QR codes and emblems but labelled neither stamp as seal and has no signature or handwriting class. Details in RESEARCH.md.

Technical findings that constrain the approach

  • Paddle CPU training on macOS arm64 (k2) runs at ~2.2 images/s: the PPLCNetV3 backbone forward alone takes 5.8 s per 32-image batch, set_num_threads has no effect, and the wheel has no MKLDNN. PyTorch MPS on the same machine trains the same network at ~70 img/s.
  • PaddleX recognition preprocessing pads to int(48 * max(320/48, w/h)) and caps the aspect width at it (floor for wide crops). A 1-px width difference flips low-confidence characters on ~40% of long lines (SVTR attention sees the padding), so the torch path mirrors it exactly (tools/deva_rec_torch/common.py, 98.8% line parity with PaddleX).
  • PaddleOCR's NRTR label encoder drops labels of max_text_length - 1 or more characters (23 with the official 25), and encode() silently skips out-of-dictionary characters.