Download ocr-tech/docs/DEVANAGARI_RESOURCES_INVENTORY.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 10.4 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/DEVANAGARI_RESOURCES_INVENTORY.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/DEVANAGARI_RESOURCES_INVENTORY.md
-
curl -L -o DEVANAGARI_RESOURCES_INVENTORY.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/DEVANAGARI_RESOURCES_INVENTORY.md
Devanagari OCR resources inventory (2026-09-24)
Scope: everything usable for a Nepali/Hindi-family Devanagari line recognizer that could
be found on our machines, our Hugging Face accounts, and public sources, checked on disk or
live on 2026-09-24. Counts are read from the artifacts, not recalled. Use decisions follow
docs/DATA_LICENSING.md; nothing here is claim-grade ground truth unless stated.
Our machines
| Where | Artifact | What it is | Status / use |
|---|---|---|---|
TRANSCEND (ocr-tech/) |
data/gorkhapatra-limbu-editions/epaper-*.pdf |
24 Gorkhapatra editions (2081-07-12 .. 2083-02-12), 493 MB, nayanepal (Preeti-family) text layer |
Primary real source. Rebuilt into chunks-v2 (below). Internal training only (owner determination 2026-07-22). |
| TRANSCEND | data/gorkhapatra-devanagari-real/ |
79,364 converted line labels from those editions; only 40,359 of the line images survived the July crash (docs/RECOVERY_LEDGER.md) |
Superseded for training by chunks-v2; surviving edition 2082-03-24 supplies the long-line test E1b. |
| TRANSCEND | outputs/newar-devanagari-silver-pack-v1-recovered/ + outputs/newar-pack-v1-normalized-2026-07-15/ |
Newar-language Devanagari lines 4,096 / 1,528 / 1,139 | NOT_ADMITTED for claims (leakage, provenance). Test split used only as a secondary diagnostic (E2), with train chunks filtered against it. |
| TRANSCEND | outputs/language-pack-silver-lines-rebuild-b{1,2}-2026-07-05.tgz |
122,882 + 184,089 page-attributed silver lines, ~50 languages (Naya Nepal pages) | Not used yet: line-level language ID and furniture filtering still required. |
| TRANSCEND | outputs/gorkhapatra-backfill-2024-03 .. 2025-07-* |
515,586 page-attributed silver lines (monthly archives) | Not used yet (same filtering requirement); largest untapped real-print pool. |
| TRANSCEND | outputs/deva-v2-reexport-2026-07-06/, spaces/limbu-ocr/models/deva-v2/ |
Our Devanagari recognizer; 143-entry dictionary with only 28 Latin letters and ASCII digits 0,1,2,3,7 | Replaced on the k2 demo by the official Paddle model on 2026-09-24. |
| TRANSCEND | data/fonts/all/devanagari/ |
289 font files | 48 families verified OFL (Google Fonts ofl/ or embedded OFL name records, incl. Annapurna SIL Nepal, Aarav). Excluded: Kalimati, Gargi, Yashomudra, Sanskrit2003 (no licence), Kohinoor (all rights reserved), Siddhanta (CC BY-NC-ND), Chandas/Uttara/Samyak (GPL), NithyaRanjanaDU (Ranjana glyphs). |
k2 (~/ocr-tech, not git) |
data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl, data/splits-mixed-v1/ |
2026-06 Nepali+English recognizer work (18,765 gauravgiri rows; mixed-v1 1,017 train / 180 eval) | Source licence unrecorded; kept out of training. |
| k2 | data/manifests-english-test/ |
1,000 NVIDIA English line crops from the earlier run | Superseded by nvidia-en-lines (below). |
| k2 | runs/paddle-recognizer-mixed-v2-bundle/, runs/hf-recognizer-* (~10 GB) |
Earlier Nepali/English recognizer experiments | Historical; not reused. |
| k2 | ~/.paddlex/official_models/ |
devanagari_PP-OCRv5_mobile_rec, PaddleOCR-VL-1.6, PP-DocLayoutV3, PP-OCRv5_server_det, PP-OCRv6_medium_det, UVDoc, orientation classifiers |
Official Devanagari model now serves ampixa.thapa-ashish.com.np. |
| cdjk | ~/deva-ft-build/ |
Build workspace: edition copies, OFL font subset, chunks-v2, synth-v1 |
Canonical build location for the 2026-09-24 fine-tune data. |
| mbp2 | ~/github/llm/ocr-tech |
Stale 2026-07-04 checkout (712 MB data/) |
Nothing newer than TRANSCEND. |
| omac | — | Offline since ~2026-08-01 | Unreachable. |
Our Hugging Face repos
| Repo | Relevant content | Use |
|---|---|---|
ampixa/ocr-tech-artifacts (private) |
deva-v2-reexport, Newar packs, aws-newar-recovery, detector datasets |
Backups of the items above. |
ampixa/nepal-mt-text (private) |
Nepali (npi) sides of MT corpora; licences mostly unrecorded or Bible text |
Weak rendering source; not used. |
Official models
| Model | Devanagari coverage | Notes |
|---|---|---|
devanagari_PP-OCRv5_mobile_rec (Paddle) |
568-entry dictionary: 128 Devanagari, all 52 Latin letters, 0-9 | Training weights (*_pretrained.pdparams, 70.5 MB) and config are published; safetensors (PaddlePaddle/devanagari_PP-OCRv5_mobile_rec_safetensors) and ONNX repos exist. |
| PP-OCRv6 | 0 Devanagari characters | Not usable for Nepali. |
| PaddleOCR-VL-1.6 (0.9B, Apache-2.0) | Lists Hindi among 109 languages | Candidate offline teacher; Nepali quality unverified. |
transformers 5.17 pp_ocrv5_mobile_rec |
PyTorch port of the same network | Used for MPS fine-tuning on k2. |
Public datasets (checked 2026-09-24)
| Dataset | Content | Licence | Decision |
|---|---|---|---|
nvidia/OCR-Synthetic-Multilingual-v1 |
Synthetic pages; en/ja/ko/ru/zh only (no Devanagari) | CC BY 4.0 | English replay: 6,802 horizontal train lines + 419 page-disjoint test lines from en/validation/validation_000.h5. |
sarvamai/indic-ocr-bench |
6,909 block-level samples, 23 languages, reviewed twice | Apache-2.0 | Future independent eval (needs page/block pipeline). |
himalaya-ai/nepali-deva-ocr-eval |
500 rows sampled from gauravgiri/nepali-ocr-dataset; the images are photographed handwriting, some labels mismatch |
none recorded | Eval-only handwriting probe (E5); never trained on. |
himalaya-ai/devanagari_ocr_dataset |
Compilation of 7 public sets, ~58 GB | Apache-2.0 tag over mixed upstream licences | Parked (size; upstream licences). |
iamkushagratomar/devanagari-ocr-synthetic-80k |
Hindi synthetic lines, 8 fonts | MIT | Optional extra font variety; not used in v1. |
Om-7387/Marathi-OCR, meharuhanzz/OCR-Bench1000-{Hindi,Marathi} |
Small Marathi/Hindi sets | Apache-2.0 / MIT | Candidate evals. |
IIIT Mozhi (darknight054/indic-mozhi-ocr mirror) |
~80k printed lines per language incl. Hindi, Marathi | No licence stated (checked CVIT page) | Excluded until CVIT confirms terms. |
| IndicSTR12 | Scene-text words, 12 languages | Disputed (reported CC BY-NC-ND) | Excluded; wrong domain. |
thegauravgiri/nepali-ocr-dataset, vishwam-101/devanagari-ocr-dataset, Titung/nepali-books-ocr-dataset-full |
Nepali/Devanagari line sets | none recorded | Excluded from training. |
Built for the 2026-09-24 fine-tune
| Set | Size | Built by |
|---|---|---|
chunks-v2 train / dev / test |
173,167 / 24,908 / 33,375 real print chunks (<= 25 codepoints), edition-disjoint (test 2082-03-24, 2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11) | tools/build_gorkhapatra_devanagari_chunks.py (pre-fix labels: carry the क्र->व्र bug) |
chunks-v4 train / dev / test (used for the deployed v3) |
125,017 / 15,570 / 21,821 chunks (<= 60 codepoints), same editions, labels converted with the `j | m->s |
synth-v1 train |
citizenship_ne 30,000; office_ne 12,000; citizenship_en 10,000; realtext 30,000 (44 OFL families) | tools/render_devanagari_form_synth.py |
synth-v1 dev / test |
800 dev (train fonts, separate seed); 1,800 test in 4 held-out families (Sura, Khula, Dekko, Sarala) | same |
Handwriting (checked 2026-09-24)
v3 is worse than the official model on handwriting (it was fine-tuned on print only), so a
handwriting fine-tune (v4) is staged. Data was downloaded on cdjk (~/hw-data/, recipes in
tools/deva_rec_torch/recipes/cdjk/) and IIIT directly on k2; sets are staged on k2 under
~/ocr-deva-ft/data/hw-train/ and hw-eval/.
| Dataset | Content | Licence | Use |
|---|---|---|---|
IIIT-HW-Dev (IIIT-INDIC-HW-WORDS Devanagari part; HF c3rl/IIIT-INDIC-HW-WORDS-Hindi, also inside Kaggle himanshukumarrajak/indic-data) |
95,430 handwritten Hindi words | research and educational use only (CVIT) | H2: 69,853 train / 2,000 dev / 2,000 test. Any model trained on it is a research model until CVIT grants permission. |
| DHCD, UCI id 389 | 92k isolated 32x32 characters, incl. 20k digits ०-९ | CC BY 4.0 | H4: 15,000 composed digit strings (dates, certificate/ward/phone numbers, amounts) from DHCD Train digits; 800 test strings from DHCD Test digits |
Kaggle sweekardahal/handwritten-labeled-dataset-for-text-recognition |
4,357 Nepali handwritten word crops | CC0 | H1: 3,187 train / 300 dev / 871 test (random word split) |
Kaggle sweekardahal/nepali-handwritten-images-for-text-detection (NHD) |
Nepali handwritten page photos with word boxes, no text | CC0 | Downloaded; unused (future handwriting detector) |
Kaggle rushdaismailaslami/handwritten-devanagari-character-and-word-dataset (IndicScan) |
835 labelled Hindi handwritten words | CC BY 4.0 | H3: test only |
Kaggle re-uploads of IIIT data under other tags (daddyshomeee MIT, sabarinathan CC BY-NC, anshmaurya420) |
IIIT data | tags on re-uploads | Treated as IIIT: research only |
Baseline (served path, codepoint CER, tools/deva_rec_torch/results/hw_baseline.json):
H1 official 40.8% vs v3 68.9%; H2 49.4% vs 60.5%; H3 25.2% vs 45.8%.
Page-level check on real letters (2026-09-24)
tools/deva_rec_torch/page_probe/: two public ocr.gov.np notices, 43 hand-transcribed
printed lines (eval-only smoke test). The worker's axis-aligned line crops give 21.4%
codepoint CER; rotated min-area crops with 6 px padding give 2.9%. PP-DocLayoutV3 found the QR
codes and emblems but labelled neither stamp as seal and has no signature or handwriting
class. Details in RESEARCH.md.
Technical findings that constrain the approach
- Paddle CPU training on macOS arm64 (k2) runs at ~2.2 images/s: the PPLCNetV3 backbone
forward alone takes 5.8 s per 32-image batch,
set_num_threadshas no effect, and the wheel has no MKLDNN. PyTorch MPS on the same machine trains the same network at ~70 img/s. - PaddleX recognition preprocessing pads to
int(48 * max(320/48, w/h))and caps the aspect width at it (floor for wide crops). A 1-px width difference flips low-confidence characters on ~40% of long lines (SVTR attention sees the padding), so the torch path mirrors it exactly (tools/deva_rec_torch/common.py, 98.8% line parity with PaddleX). - PaddleOCR's NRTR label encoder drops labels of
max_text_length - 1or more characters (23 with the official 25), andencode()silently skips out-of-dictionary characters.