ocr-projects-archive / ocr-tech /docs /DEVANAGARI_RESOURCES_INVENTORY.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
10.4 kB
# Devanagari OCR resources inventory (2026-09-24)
Scope: everything usable for a Nepali/Hindi-family Devanagari line recognizer that could
be found on our machines, our Hugging Face accounts, and public sources, checked on disk or
live on 2026-09-24. Counts are read from the artifacts, not recalled. Use decisions follow
`docs/DATA_LICENSING.md`; nothing here is claim-grade ground truth unless stated.
## Our machines
| Where | Artifact | What it is | Status / use |
|---|---|---|---|
| TRANSCEND (`ocr-tech/`) | `data/gorkhapatra-limbu-editions/epaper-*.pdf` | 24 Gorkhapatra editions (2081-07-12 .. 2083-02-12), 493 MB, `nayanepal` (Preeti-family) text layer | Primary real source. Rebuilt into `chunks-v2` (below). Internal training only (owner determination 2026-07-22). |
| TRANSCEND | `data/gorkhapatra-devanagari-real/` | 79,364 converted line labels from those editions; only 40,359 of the line images survived the July crash (`docs/RECOVERY_LEDGER.md`) | Superseded for training by `chunks-v2`; surviving edition 2082-03-24 supplies the long-line test E1b. |
| TRANSCEND | `outputs/newar-devanagari-silver-pack-v1-recovered/` + `outputs/newar-pack-v1-normalized-2026-07-15/` | Newar-language Devanagari lines 4,096 / 1,528 / 1,139 | `NOT_ADMITTED` for claims (leakage, provenance). Test split used only as a secondary diagnostic (E2), with train chunks filtered against it. |
| TRANSCEND | `outputs/language-pack-silver-lines-rebuild-b{1,2}-2026-07-05.tgz` | 122,882 + 184,089 page-attributed silver lines, ~50 languages (Naya Nepal pages) | Not used yet: line-level language ID and furniture filtering still required. |
| TRANSCEND | `outputs/gorkhapatra-backfill-2024-03 .. 2025-07-*` | 515,586 page-attributed silver lines (monthly archives) | Not used yet (same filtering requirement); largest untapped real-print pool. |
| TRANSCEND | `outputs/deva-v2-reexport-2026-07-06/`, `spaces/limbu-ocr/models/deva-v2/` | Our Devanagari recognizer; 143-entry dictionary with only 28 Latin letters and ASCII digits 0,1,2,3,7 | Replaced on the k2 demo by the official Paddle model on 2026-09-24. |
| TRANSCEND | `data/fonts/all/devanagari/` | 289 font files | 48 families verified OFL (Google Fonts `ofl/` or embedded OFL name records, incl. Annapurna SIL Nepal, Aarav). Excluded: Kalimati, Gargi, Yashomudra, Sanskrit2003 (no licence), Kohinoor (all rights reserved), Siddhanta (CC BY-NC-ND), Chandas/Uttara/Samyak (GPL), NithyaRanjanaDU (Ranjana glyphs). |
| k2 (`~/ocr-tech`, not git) | `data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl`, `data/splits-mixed-v1/` | 2026-06 Nepali+English recognizer work (18,765 gauravgiri rows; mixed-v1 1,017 train / 180 eval) | Source licence unrecorded; kept out of training. |
| k2 | `data/manifests-english-test/` | 1,000 NVIDIA English line crops from the earlier run | Superseded by `nvidia-en-lines` (below). |
| k2 | `runs/paddle-recognizer-mixed-v2-bundle/`, `runs/hf-recognizer-*` (~10 GB) | Earlier Nepali/English recognizer experiments | Historical; not reused. |
| k2 | `~/.paddlex/official_models/` | `devanagari_PP-OCRv5_mobile_rec`, `PaddleOCR-VL-1.6`, `PP-DocLayoutV3`, `PP-OCRv5_server_det`, `PP-OCRv6_medium_det`, `UVDoc`, orientation classifiers | Official Devanagari model now serves `ampixa.thapa-ashish.com.np`. |
| cdjk | `~/deva-ft-build/` | Build workspace: edition copies, OFL font subset, `chunks-v2`, `synth-v1` | Canonical build location for the 2026-09-24 fine-tune data. |
| mbp2 | `~/github/llm/ocr-tech` | Stale 2026-07-04 checkout (712 MB `data/`) | Nothing newer than TRANSCEND. |
| omac | — | Offline since ~2026-08-01 | Unreachable. |
## Our Hugging Face repos
| Repo | Relevant content | Use |
|---|---|---|
| `ampixa/ocr-tech-artifacts` (private) | `deva-v2-reexport`, Newar packs, `aws-newar-recovery`, detector datasets | Backups of the items above. |
| `ampixa/nepal-mt-text` (private) | Nepali (`npi`) sides of MT corpora; licences mostly unrecorded or Bible text | Weak rendering source; not used. |
## Official models
| Model | Devanagari coverage | Notes |
|---|---|---|
| `devanagari_PP-OCRv5_mobile_rec` (Paddle) | 568-entry dictionary: 128 Devanagari, all 52 Latin letters, 0-9 | Training weights (`*_pretrained.pdparams`, 70.5 MB) and config are published; safetensors (`PaddlePaddle/devanagari_PP-OCRv5_mobile_rec_safetensors`) and ONNX repos exist. |
| PP-OCRv6 | 0 Devanagari characters | Not usable for Nepali. |
| PaddleOCR-VL-1.6 (0.9B, Apache-2.0) | Lists Hindi among 109 languages | Candidate offline teacher; Nepali quality unverified. |
| `transformers` 5.17 `pp_ocrv5_mobile_rec` | PyTorch port of the same network | Used for MPS fine-tuning on k2. |
## Public datasets (checked 2026-09-24)
| Dataset | Content | Licence | Decision |
|---|---|---|---|
| `nvidia/OCR-Synthetic-Multilingual-v1` | Synthetic pages; en/ja/ko/ru/zh only (no Devanagari) | CC BY 4.0 | English replay: 6,802 horizontal train lines + 419 page-disjoint test lines from `en/validation/validation_000.h5`. |
| `sarvamai/indic-ocr-bench` | 6,909 block-level samples, 23 languages, reviewed twice | Apache-2.0 | Future independent eval (needs page/block pipeline). |
| `himalaya-ai/nepali-deva-ocr-eval` | 500 rows sampled from `gauravgiri/nepali-ocr-dataset`; the images are photographed **handwriting**, some labels mismatch | none recorded | Eval-only handwriting probe (E5); never trained on. |
| `himalaya-ai/devanagari_ocr_dataset` | Compilation of 7 public sets, ~58 GB | Apache-2.0 tag over mixed upstream licences | Parked (size; upstream licences). |
| `iamkushagratomar/devanagari-ocr-synthetic-80k` | Hindi synthetic lines, 8 fonts | MIT | Optional extra font variety; not used in v1. |
| `Om-7387/Marathi-OCR`, `meharuhanzz/OCR-Bench1000-{Hindi,Marathi}` | Small Marathi/Hindi sets | Apache-2.0 / MIT | Candidate evals. |
| IIIT Mozhi (`darknight054/indic-mozhi-ocr` mirror) | ~80k printed lines per language incl. Hindi, Marathi | No licence stated (checked CVIT page) | Excluded until CVIT confirms terms. |
| IndicSTR12 | Scene-text words, 12 languages | Disputed (reported CC BY-NC-ND) | Excluded; wrong domain. |
| `thegauravgiri/nepali-ocr-dataset`, `vishwam-101/devanagari-ocr-dataset`, `Titung/nepali-books-ocr-dataset-full` | Nepali/Devanagari line sets | none recorded | Excluded from training. |
## Built for the 2026-09-24 fine-tune
| Set | Size | Built by |
|---|---|---|
| `chunks-v2` train / dev / test | 173,167 / 24,908 / 33,375 real print chunks (<= 25 codepoints), edition-disjoint (test 2082-03-24, 2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11) | `tools/build_gorkhapatra_devanagari_chunks.py` (pre-fix labels: carry the क्र->व्र bug) |
| `chunks-v4` train / dev / test (used for the deployed v3) | 125,017 / 15,570 / 21,821 chunks (<= 60 codepoints), same editions, labels converted with the `j|m` -> `s|` क्र fix (3,425 spans corrected, 2,801 previously rejected spans recovered) | same tool with `RAW_FIXES` |
| `synth-v1` train | citizenship_ne 30,000; office_ne 12,000; citizenship_en 10,000; realtext 30,000 (44 OFL families) | `tools/render_devanagari_form_synth.py` |
| `synth-v1` dev / test | 800 dev (train fonts, separate seed); 1,800 test in 4 held-out families (Sura, Khula, Dekko, Sarala) | same |
## Handwriting (checked 2026-09-24)
v3 is worse than the official model on handwriting (it was fine-tuned on print only), so a
handwriting fine-tune (v4) is staged. Data was downloaded on cdjk (`~/hw-data/`, recipes in
`tools/deva_rec_torch/recipes/cdjk/`) and IIIT directly on k2; sets are staged on k2 under
`~/ocr-deva-ft/data/hw-train/` and `hw-eval/`.
| Dataset | Content | Licence | Use |
|---|---|---|---|
| IIIT-HW-Dev (IIIT-INDIC-HW-WORDS Devanagari part; HF `c3rl/IIIT-INDIC-HW-WORDS-Hindi`, also inside Kaggle `himanshukumarrajak/indic-data`) | 95,430 handwritten Hindi words | research and educational use only (CVIT) | H2: 69,853 train / 2,000 dev / 2,000 test. **Any model trained on it is a research model** until CVIT grants permission. |
| DHCD, UCI id 389 | 92k isolated 32x32 characters, incl. 20k digits ०-९ | CC BY 4.0 | H4: 15,000 composed digit strings (dates, certificate/ward/phone numbers, amounts) from DHCD Train digits; 800 test strings from DHCD Test digits |
| Kaggle `sweekardahal/handwritten-labeled-dataset-for-text-recognition` | 4,357 Nepali handwritten word crops | CC0 | H1: 3,187 train / 300 dev / 871 test (random word split) |
| Kaggle `sweekardahal/nepali-handwritten-images-for-text-detection` (NHD) | Nepali handwritten page photos with word boxes, no text | CC0 | Downloaded; unused (future handwriting detector) |
| Kaggle `rushdaismailaslami/handwritten-devanagari-character-and-word-dataset` (IndicScan) | 835 labelled Hindi handwritten words | CC BY 4.0 | H3: test only |
| Kaggle re-uploads of IIIT data under other tags (`daddyshomeee` MIT, `sabarinathan` CC BY-NC, `anshmaurya420`) | IIIT data | tags on re-uploads | Treated as IIIT: research only |
Baseline (served path, codepoint CER, `tools/deva_rec_torch/results/hw_baseline.json`):
H1 official 40.8% vs v3 68.9%; H2 49.4% vs 60.5%; H3 25.2% vs 45.8%.
## Page-level check on real letters (2026-09-24)
`tools/deva_rec_torch/page_probe/`: two public ocr.gov.np notices, 43 hand-transcribed
printed lines (eval-only smoke test). The worker's axis-aligned line crops give 21.4%
codepoint CER; rotated min-area crops with 6 px padding give 2.9%. PP-DocLayoutV3 found the QR
codes and emblems but labelled neither stamp as `seal` and has no signature or handwriting
class. Details in `RESEARCH.md`.
## Technical findings that constrain the approach
- Paddle CPU training on macOS arm64 (k2) runs at ~2.2 images/s: the PPLCNetV3 backbone
forward alone takes 5.8 s per 32-image batch, `set_num_threads` has no effect, and the
wheel has no MKLDNN. PyTorch MPS on the same machine trains the same network at ~70 img/s.
- PaddleX recognition preprocessing pads to `int(48 * max(320/48, w/h))` and caps the
aspect width at it (floor for wide crops). A 1-px width difference flips low-confidence
characters on ~40% of long lines (SVTR attention sees the padding), so the torch path
mirrors it exactly (`tools/deva_rec_torch/common.py`, 98.8% line parity with PaddleX).
- PaddleOCR's NRTR label encoder drops labels of `max_text_length - 1` or more characters
(23 with the official 25), and `encode()` silently skips out-of-dictionary characters.