# Nepal Multilingual Document OCR Goal This is the research target after the steering decision on 2026-06-16. The project is now a PaddleOCR-centered multilingual document OCR system for languages used in Nepal, not a narrow Nepali/English OCR experiment. ## Goal Build and validate a PaddleOCR-based document OCR engine for at least ten languages spoken in Nepal, with Limbu first, that supports: - page/PDF/image parsing - text OCR - reading order - table reconstruction - figure and caption extraction - Markdown export - structured JSON export - selective PaddleOCR-VL/VLM fallback for hard pages The claim is valid only if the engine beats strong baselines on predefined real and held-out evaluations. Synthetic-only results are not enough. ## Research Claim Shape The paper should not claim "best OCR engine overall." The defensible claim is: > A script-aware PaddleOCR adaptation framework can substantially improve > document OCR and structured extraction for low-resource Nepal languages, > including scripts not covered well by stock OCR systems, while preserving > Nepali/English performance and using selective VLM fallback only when the > local pipeline is uncertain. The core contribution is the system and validation recipe: 1. Language/script registry for Nepal document OCR. 2. Gorkhapatra/Naya Nepal real-publication validation pack. 3. Unicode/font/renderability audit before synthesis. 4. Real-text synthetic data generation for lines, tables, captions, and pages. 5. PaddleOCR recognizer fine-tuning by script family. 6. PP-Structure-style document extraction. 7. Claim review that rejects weak SOTA statements. ## Primary Engine Direction Use PaddleOCR as the engine family: - **PP-OCRv6** for the modern small/local OCR baseline and fine-tuning target. - **PP-StructureV3** for layout, reading order, tables, and document structure. - **PaddleOCR-VL-1.6** as fallback/teacher for charts, figures, hard layouts, seals, and low-confidence pages. Do not route everything to a VLM and call that the engine. The local PaddleOCR path must be the default path; fallback usage must be measured and reported. ## First Ten Language Targets The first target set should be driven by actual Gorkhapatra/Naya Nepal availability plus local corpus availability. OCR support is tracked by script, Unicode, fonts, and evaluation data, not by language name alone. | Priority | Language | First script target | Why first | Notes | | --- | --- | --- | --- | --- | | 1 | Limbu/Yakthung | Limbu/Sirijonga, U+1900..U+194F | Has own Unicode block and local Limbu resources | First non-Devanagari research target | | 2 | Nepali | Devanagari | Core language and baseline | Must not regress | | 3 | English | Latin | Mixed documents and forms | Must not regress | | 4 | Tamang | Devanagari first; Tamyig audit later | Local NepTam corpus and dictionary | Start with Devanagari Tamang | | 5 | Magar | Devanagari first | Gorkhapatra item confirmed; local dictionary | Own/community script requires separate audit | | 6 | Wambule | Devanagari first | Gorkhapatra item confirmed | Rai-family validation candidate | | 7 | Bantawa/Rai | Devanagari or Kirat Rai depending source | Gorkhapatra item confirmed; Rai resources on remote/local projects | Kirat Rai Unicode support is newer; font audit required | | 8 | Kulung | Devanagari first | Gorkhapatra item confirmed | Rai-family validation candidate | | 9 | Yakkha | Devanagari first | Gorkhapatra item confirmed | Rai-family validation candidate | | 10 | Tharu/Rana Tharu | Devanagari first | Gorkhapatra item confirmed | Multiple varieties; keep labels explicit | Additional Gorkhapatra-visible candidates include Sanskrit, Magahi, Bajjika, Sunuwar, Danuwar, Majhi, Lhomi, Bhojpuri, Kisan, Thami, Sherpa, Achhami, Bote, Rajbanshi, Hayu, Santhali, Dhimal, Bhote, Uraon, Kumal, Gurung, Khash Jumli, and Meche. These should enter the registry after script/corpus/font checks. ## Source Findings So Far ### Gorkhapatra Naya Nepal The category page is useful for language-specific validation: - `https://gorkhapatraonline.com/categories/new-nepal` - `https://gorkhapatraonline.com/categories/new-nepal?page=2` Confirmed visible language-specific entries include Magar, Wambule, Rana Tharu, Magahi, Bajjika, Sunuwar, Danuwar, Majhi, Lhomi, Bhojpuri, Kisan, Thami, Sherpa, Achhami, Bote, Rajbanshi, Hayu, Santhali, Kulung, Yakkha, Meche, Khash Jumli, Dhimal, Tharu, Bhote, Uraon, Bantawa, Kumal, Gurung, and others. The blog/category pages expose image URLs and article URLs. They are useful for real-world OCR samples, but they should not be trusted as clean text references without manual or tool-assisted verification. ### Gorkhapatra Epaper The epaper list endpoint exposes viewer links: ```text https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-09&pdfdaterange2=2026-06-16&slug=gorkhapatra ``` The visible links look like: ```text https://epaper.gorkhapatraonline.com/pdf?file=/uploads/file/2026/6/gorkhapatra/2026-06-14-11-49-50-2083-03-01.pdf ``` That `pdf?...` URL returns a PDF.js viewer shell. The real PDF is downloaded by fetching the file parameter directly: ```text https://epaper.gorkhapatraonline.com/uploads/file/2026/6/gorkhapatra/2026-06-14-11-49-50-2083-03-01.pdf ``` On `k2`, this was verified as a real PDF: ```text /Users/k2/ocr-tech/outputs/gorkhapatra-source-audit/sample.pdf PDF document, version 1.6 size: 48M ``` The `k2` audit produced: ```text /Users/k2/ocr-tech/outputs/gorkhapatra-source-audit/audit.txt /Users/k2/ocr-tech/outputs/gorkhapatra-source-audit/new-nepal-language-items.txt /Users/k2/ocr-tech/outputs/gorkhapatra-source-audit/epaper-assets.txt /Users/k2/ocr-tech/outputs/gorkhapatra-source-audit/sample.pdf ``` `k2` does not currently expose `pdfinfo` or `pdftotext`. `pymupdf` was installed into `/Users/k2/ocr-tech/.venv` on 2026-06-16 as the immediate PDF audit tool. The sample epaper PDF opens successfully with PyMuPDF: ```text page_count: 26 total extracted text chars: 309924 preview image: outputs/gorkhapatra-source-audit/sample-page-1-preview.jpg audit: outputs/gorkhapatra-source-audit/sample-pymupdf-audit.json text: outputs/gorkhapatra-source-audit/sample-pymupdf-text.txt ``` The extracted text layer is not reliable reference text. The first-page preview contains mojibake/transliterated-looking output for Nepali content rather than clean Unicode Devanagari. Therefore the epaper PDFs are useful as real page images/layout sources, but their embedded text layer must be audited per file and must not be treated as ground truth without verification. ### Local And Remote Corpus Hints Local machine: - `~/github/llm/ltk/tamang`: NepTam train/test, NepTam80K, dictionary, TTS lexicon. - `~/github/llm/ltk/limbu`: Limbu dictionary, Pangloss IGT, Limbu G2P. - `~/github/llm/ltk/magar`: Western Magar dictionary and resource report. Remote `cdjk@100.110.148.43`: - `/Volumes/TRANSCEND` has about 2.0 TiB free. - `~/github/llm` contains additional language projects: `dhimal-toolkit`, `gurung-language-toolkit`, `limbu-speech-toolkit`, `magar-toolkit`, `rai-speech-toolkit`, `nllb-better`, `nepa-newa-text-frontend`, `new-place`, and `loksewa.zip`. - `nllb-better` has Rai-family generated/validation artifacts and a language registry that should be inspected before building the OCR registry. Reproducible inventory gate: - `ocrtech audit-synthesis-resources` records file counts, suffix counts, sample hashes, and Unicode block hints before any converter is written. - The first local LTK audit is documented in `docs/SYNTHESIS_RESOURCE_AUDIT.md`. - A source-only copy was synced to `/tmp/ocr-tech` on `cdjk@100.110.148.43`; remote toolkit and `/Volumes/TRANSCEND` audits are documented in `docs/SYNTHESIS_RESOURCE_AUDIT.md`. - The first schema-specific converter, `ocrtech prepare-limbu-limdic-text`, emits source-hashed Limbu Limdic synthesis-text JSONL after verifying actual JSON/TSV/Firebase schemas. - `ocrtech render-synthesis-text-lines` turns verified synthesis-text JSONL into PaddleOCR-style line PNGs and label files with font SHA provenance. - `ocrtech audit-rendered-synthesis-lines` verifies rendered synthetic line manifests before training bundles: image hashes, PaddleOCR labels, source schema provenance, font provenance when required, and `claim_evidence_eligible=false`. ## Data Plan ### Real Validation Data Use Gorkhapatra/Naya Nepal as the central real-publication validation source. Create a scraper that emits: - article URL - article title - language label extracted from title - article image URLs - epaper date - epaper PDF URL - direct PDF file URL - image/PDF hashes - source timestamp - license/terms note - manual-review status Then create a hand-labeled validation pack: - at least 10 languages - at least 20 pages or page crops per language for v1 if available - Limbu set first - no synthetic samples in the final real validation pack - references include text, line boxes, reading order, table cells, captions, and Markdown where applicable ### Synthetic Training Data Use real text when synthesizing: - Limbu dictionary/Pangloss/IGT text first. - Tamang NepTam and dictionary text. - Devanagari text from Gorkhapatra/Loksewa/government-like documents where licensing permits. - Rai-family text from verified local/remote corpora. Synthetic generation must produce: - line crops - word crops - table cell crops - captions - full pages - degraded scan/photo variants - PaddleOCR recognition labels - document JSON/Markdown references Every synthetic sample must carry: - language - script - Unicode ranges observed - font - source text hash - rendered image hash - template id - degradation id - train/eval split id ### Handwriting Synthesis Handwriting is not a v1 blocker. It can be a later ablation. Possible direction: - Use available fonts and handwritten-style fonts for cheap diversity. - Explore diffusion/DM-based handwriting generation only after print OCR is stable. - For Limbu, do not synthesize handwriting until we have real handwritten or community-reviewed glyph examples. Otherwise we risk generating plausible but wrong orthography. If we explore One-DM or related diffusion methods, it should be framed as optional augmentation, not the main claim. ## Validation Gates ### Gate 0: Source Audit Pass only if: - Gorkhapatra article image URLs are downloadable. - Epaper direct PDFs are downloadable. - PDF text/image extraction path is implemented and audited. - Every source artifact is hashed. - Source license/terms risk is recorded. ### Gate 1: Language Registry Pass only if every target language has: - language name - ISO code if known - script name - Unicode range - font list - corpus source - renderability audit - normalization policy - PaddleOCR dictionary path - evaluation availability ### Gate 2: Limbu First Pass only if: - Limbu Unicode rendering works with selected fonts. - Limbu PaddleOCR dictionary exists. - Limbu synthetic crop/page generator emits valid labels. - Limbu held-out real or manually reviewed eval exists. - Stock baselines are recorded before fine-tuning. - Claim-grade recognizer bundles use `--require-eval-real`; rendered synthetic Limbu rows may train/ablate but must not appear in held-out eval evidence. - Gorkhapatra language-page candidates are promoted through `ocrtech finalize-gorkhapatra-language-page-review` only after accepted review rows have verified structured reference JSON with text, reading order, tables, and figures. - `ocrtech audit-gorkhapatra-language-page-review` must pass before finalization: exactly one page candidate is accepted per publication/language/date group, and accepted rows point to reference files. Use `--require-verified-references` for the final pre-claim gate. - `ocrtech prepare-gorkhapatra-language-page-reference-templates` creates draft structured references for reviewers without making them claim-eligible. ### Gate 3: PaddleOCR Baselines Run before any claim: - stock PP-OCRv6 - stock PP-OCRv5 if available - Tesseract where traineddata exists - Surya - GLM-OCR - PaddleOCR-VL-1.6 ### Gate 4: Fine-Tuned Candidate The candidate must beat baselines on: - CER/WER per language - strict and normalized Unicode CER - exact-line accuracy - reading order - table text and structure - figure/caption detection - Markdown reconstruction - JSON artifact completeness English and Nepali must not regress versus the chosen stock baseline. ### Gate 5: Claim Review The paper claim is allowed only if: - all public/real validation rows have complete artifacts - fallback usage is reported separately - synthetic-only improvements are not used as SOTA evidence - failures are included in error analysis - a reproducibility bundle can rerun the benchmark ## Immediate Implementation Steps 1. Add `ocrtech audit-gorkhapatra-source`. **Implemented:** the command audits Naya Nepal article images and Gorkhapatra epaper PDF links, resolves PDF.js viewer URLs to direct `/uploads/file/...pdf` URLs, optionally downloads and hashes assets, and records PyMuPDF text-layer metadata when available. 2. Add `ocrtech prepare-gorkhapatra-pack`. **Implemented:** the command turns a source-audit JSON file into a pending-reference validation manifest, copies audited article images, renders audited epaper PDF pages when PyMuPDF is available, writes placeholder reference JSON files, and marks samples as `reference_status=pending_manual_label` and `claim_evidence_eligible=false` until references are verified. 3. Add `languages/nepal-v1.json` registry with Limbu first and at least ten target languages. **Implemented:** the registry tracks Limbu/Sirijonga first, more than ten Nepal-language targets, English as a non-counted regression language, script/Unicode ranges, font candidates, corpus sources, Gorkhapatra labels, validation status, and PaddleOCR dictionary targets. The `ocrtech audit-language-registry` command validates the registry and writes JSON/Markdown audit artifacts. 4. Add renderability audit for Unicode ranges and fonts. **Implemented:** the `ocrtech audit-font-renderability` command reads TTF/OTF/TTC cmap tables, checks registered font candidates against each script's Unicode ranges, and writes JSON/Markdown evidence. Raster sample rendering is supported when Pillow is installed. Current cmap evidence verifies Limbu, Devanagari, Santhali/Ol Chiki, and Latin coverage when the registered runtime font assets are present. 5. Add target-language readiness audit. **Implemented:** the `ocrtech audit-language-readiness` command combines `languages/nepal-v1.json` with passed `synthesis-text-manifest-audit.json` evidence and writes a per-language checklist. Current run: ```bash uv run ocrtech audit-language-readiness \ languages/nepal-v1.json \ --out outputs/language-readiness-audit-2026-06-16 \ --synthesis-text-audit outputs/limbu-unicode-text-smoke-audit-2026-06-16/synthesis-text-manifest-audit.json \ --synthesis-text-audit outputs/tamang-text-smoke-audit-2026-06-16/synthesis-text-manifest-audit.json \ --synthesis-text-audit outputs/magar-text-smoke-audit-2026-06-16/synthesis-text-manifest-audit.json \ --synthesis-text-audit outputs/remote-bible-brain-readiness-2026-06-16/synthesis-text-manifest-audit.json \ --synthesis-text-audit outputs/remote-bible-brain-readiness-extra-2026-06-16/synthesis-text-manifest-audit.json ``` Result: `passed=True target_languages=17 synthesis_ready=11/1`. Limbu is synthesis-text ready with 200 audited samples, Tamang is synthesis-text ready with 300 audited Devanagari samples, and Magar is synthesis-text ready with 200 audited lexicon-derived samples. Remote Bible Brain adds 80 audited Devanagari samples each for Nepali, Rana Tharu, and Kulung, plus 60 audited samples each for Gurung, Dhimal, Sunuwar, Rajbanshi, and Sherpa. Every other target language is still a source-candidate or registry-only item until a converter and `audit-synthesis-text-manifest` run exists. 6. Add Limbu synthetic line/page generator. 7. Add PP-OCRv6 stock baseline runner. 8. Build the first validation pack through Gorkhapatra/Naya Nepal publication date alignment: extract Naya Nepal card dates, match the same-date Gorkhapatra epaper PDF, locate the full `भाषा पृष्ठ` page, render that PDF page, and keep it pending manual reference labeling. See [Gorkhapatra Language Page Alignment](GORKHAPATRA_LANGUAGE_PAGE_ALIGNMENT.md). 9. Keep synthesis resource audits current on `cdjk@100.110.148.43`, especially the 4TB SSD paths `/Volumes/TRANSCEND` and `~/github/llm`, before creating real-text synthesis manifests. 10. Use `audit-synthesis-resources` output as the inventory gate; then build schema-specific converters for Limbu, Tamang, Magar, and remote Bible Brain resources only after actual fields are verified. ## External References - PaddleOCR docs: - PaddleOCR repo: - PP-StructureV3 docs: - PaddleOCR-VL-1.6 docs: - Gorkhapatra Naya Nepal category: - Gorkhapatra Naya Nepal publication: - Gorkhapatra epaper list example: - Unicode Limbu chart: - Kirat Rai Unicode proposal: