ocr-projects-archive / ocr-tech /docs /NEPAL_MULTILINGUAL_OCR_GOAL.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
18 kB

Nepal Multilingual Document OCR Goal

This is the research target after the steering decision on 2026-06-16. The project is now a PaddleOCR-centered multilingual document OCR system for languages used in Nepal, not a narrow Nepali/English OCR experiment.

Goal

Build and validate a PaddleOCR-based document OCR engine for at least ten languages spoken in Nepal, with Limbu first, that supports:

  • page/PDF/image parsing
  • text OCR
  • reading order
  • table reconstruction
  • figure and caption extraction
  • Markdown export
  • structured JSON export
  • selective PaddleOCR-VL/VLM fallback for hard pages

The claim is valid only if the engine beats strong baselines on predefined real and held-out evaluations. Synthetic-only results are not enough.

Research Claim Shape

The paper should not claim "best OCR engine overall." The defensible claim is:

A script-aware PaddleOCR adaptation framework can substantially improve document OCR and structured extraction for low-resource Nepal languages, including scripts not covered well by stock OCR systems, while preserving Nepali/English performance and using selective VLM fallback only when the local pipeline is uncertain.

The core contribution is the system and validation recipe:

  1. Language/script registry for Nepal document OCR.
  2. Gorkhapatra/Naya Nepal real-publication validation pack.
  3. Unicode/font/renderability audit before synthesis.
  4. Real-text synthetic data generation for lines, tables, captions, and pages.
  5. PaddleOCR recognizer fine-tuning by script family.
  6. PP-Structure-style document extraction.
  7. Claim review that rejects weak SOTA statements.

Primary Engine Direction

Use PaddleOCR as the engine family:

  • PP-OCRv6 for the modern small/local OCR baseline and fine-tuning target.
  • PP-StructureV3 for layout, reading order, tables, and document structure.
  • PaddleOCR-VL-1.6 as fallback/teacher for charts, figures, hard layouts, seals, and low-confidence pages.

Do not route everything to a VLM and call that the engine. The local PaddleOCR path must be the default path; fallback usage must be measured and reported.

First Ten Language Targets

The first target set should be driven by actual Gorkhapatra/Naya Nepal availability plus local corpus availability. OCR support is tracked by script, Unicode, fonts, and evaluation data, not by language name alone.

Priority Language First script target Why first Notes
1 Limbu/Yakthung Limbu/Sirijonga, U+1900..U+194F Has own Unicode block and local Limbu resources First non-Devanagari research target
2 Nepali Devanagari Core language and baseline Must not regress
3 English Latin Mixed documents and forms Must not regress
4 Tamang Devanagari first; Tamyig audit later Local NepTam corpus and dictionary Start with Devanagari Tamang
5 Magar Devanagari first Gorkhapatra item confirmed; local dictionary Own/community script requires separate audit
6 Wambule Devanagari first Gorkhapatra item confirmed Rai-family validation candidate
7 Bantawa/Rai Devanagari or Kirat Rai depending source Gorkhapatra item confirmed; Rai resources on remote/local projects Kirat Rai Unicode support is newer; font audit required
8 Kulung Devanagari first Gorkhapatra item confirmed Rai-family validation candidate
9 Yakkha Devanagari first Gorkhapatra item confirmed Rai-family validation candidate
10 Tharu/Rana Tharu Devanagari first Gorkhapatra item confirmed Multiple varieties; keep labels explicit

Additional Gorkhapatra-visible candidates include Sanskrit, Magahi, Bajjika, Sunuwar, Danuwar, Majhi, Lhomi, Bhojpuri, Kisan, Thami, Sherpa, Achhami, Bote, Rajbanshi, Hayu, Santhali, Dhimal, Bhote, Uraon, Kumal, Gurung, Khash Jumli, and Meche. These should enter the registry after script/corpus/font checks.

Source Findings So Far

Gorkhapatra Naya Nepal

The category page is useful for language-specific validation:

  • https://gorkhapatraonline.com/categories/new-nepal
  • https://gorkhapatraonline.com/categories/new-nepal?page=2

Confirmed visible language-specific entries include Magar, Wambule, Rana Tharu, Magahi, Bajjika, Sunuwar, Danuwar, Majhi, Lhomi, Bhojpuri, Kisan, Thami, Sherpa, Achhami, Bote, Rajbanshi, Hayu, Santhali, Kulung, Yakkha, Meche, Khash Jumli, Dhimal, Tharu, Bhote, Uraon, Bantawa, Kumal, Gurung, and others.

The blog/category pages expose image URLs and article URLs. They are useful for real-world OCR samples, but they should not be trusted as clean text references without manual or tool-assisted verification.

Gorkhapatra Epaper

The epaper list endpoint exposes viewer links:

https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-09&pdfdaterange2=2026-06-16&slug=gorkhapatra

The visible links look like:

https://epaper.gorkhapatraonline.com/pdf?file=/uploads/file/2026/6/gorkhapatra/2026-06-14-11-49-50-2083-03-01.pdf

That pdf?... URL returns a PDF.js viewer shell. The real PDF is downloaded by fetching the file parameter directly:

https://epaper.gorkhapatraonline.com/uploads/file/2026/6/gorkhapatra/2026-06-14-11-49-50-2083-03-01.pdf

On k2, this was verified as a real PDF:

/Users/k2/ocr-tech/outputs/gorkhapatra-source-audit/sample.pdf
PDF document, version 1.6
size: 48M

The k2 audit produced:

/Users/k2/ocr-tech/outputs/gorkhapatra-source-audit/audit.txt
/Users/k2/ocr-tech/outputs/gorkhapatra-source-audit/new-nepal-language-items.txt
/Users/k2/ocr-tech/outputs/gorkhapatra-source-audit/epaper-assets.txt
/Users/k2/ocr-tech/outputs/gorkhapatra-source-audit/sample.pdf

k2 does not currently expose pdfinfo or pdftotext. pymupdf was installed into /Users/k2/ocr-tech/.venv on 2026-06-16 as the immediate PDF audit tool. The sample epaper PDF opens successfully with PyMuPDF:

page_count: 26
total extracted text chars: 309924
preview image: outputs/gorkhapatra-source-audit/sample-page-1-preview.jpg
audit: outputs/gorkhapatra-source-audit/sample-pymupdf-audit.json
text: outputs/gorkhapatra-source-audit/sample-pymupdf-text.txt

The extracted text layer is not reliable reference text. The first-page preview contains mojibake/transliterated-looking output for Nepali content rather than clean Unicode Devanagari. Therefore the epaper PDFs are useful as real page images/layout sources, but their embedded text layer must be audited per file and must not be treated as ground truth without verification.

Local And Remote Corpus Hints

Local machine:

  • ~/github/llm/ltk/tamang: NepTam train/test, NepTam80K, dictionary, TTS lexicon.
  • ~/github/llm/ltk/limbu: Limbu dictionary, Pangloss IGT, Limbu G2P.
  • ~/github/llm/ltk/magar: Western Magar dictionary and resource report.

Remote cdjk@100.110.148.43:

  • /Volumes/TRANSCEND has about 2.0 TiB free.
  • ~/github/llm contains additional language projects: dhimal-toolkit, gurung-language-toolkit, limbu-speech-toolkit, magar-toolkit, rai-speech-toolkit, nllb-better, nepa-newa-text-frontend, new-place, and loksewa.zip.
  • nllb-better has Rai-family generated/validation artifacts and a language registry that should be inspected before building the OCR registry.

Reproducible inventory gate:

  • ocrtech audit-synthesis-resources records file counts, suffix counts, sample hashes, and Unicode block hints before any converter is written.
  • The first local LTK audit is documented in docs/SYNTHESIS_RESOURCE_AUDIT.md.
  • A source-only copy was synced to /tmp/ocr-tech on cdjk@100.110.148.43; remote toolkit and /Volumes/TRANSCEND audits are documented in docs/SYNTHESIS_RESOURCE_AUDIT.md.
  • The first schema-specific converter, ocrtech prepare-limbu-limdic-text, emits source-hashed Limbu Limdic synthesis-text JSONL after verifying actual JSON/TSV/Firebase schemas.
  • ocrtech render-synthesis-text-lines turns verified synthesis-text JSONL into PaddleOCR-style line PNGs and label files with font SHA provenance.
  • ocrtech audit-rendered-synthesis-lines verifies rendered synthetic line manifests before training bundles: image hashes, PaddleOCR labels, source schema provenance, font provenance when required, and claim_evidence_eligible=false.

Data Plan

Real Validation Data

Use Gorkhapatra/Naya Nepal as the central real-publication validation source.

Create a scraper that emits:

  • article URL
  • article title
  • language label extracted from title
  • article image URLs
  • epaper date
  • epaper PDF URL
  • direct PDF file URL
  • image/PDF hashes
  • source timestamp
  • license/terms note
  • manual-review status

Then create a hand-labeled validation pack:

  • at least 10 languages
  • at least 20 pages or page crops per language for v1 if available
  • Limbu set first
  • no synthetic samples in the final real validation pack
  • references include text, line boxes, reading order, table cells, captions, and Markdown where applicable

Synthetic Training Data

Use real text when synthesizing:

  • Limbu dictionary/Pangloss/IGT text first.
  • Tamang NepTam and dictionary text.
  • Devanagari text from Gorkhapatra/Loksewa/government-like documents where licensing permits.
  • Rai-family text from verified local/remote corpora.

Synthetic generation must produce:

  • line crops
  • word crops
  • table cell crops
  • captions
  • full pages
  • degraded scan/photo variants
  • PaddleOCR recognition labels
  • document JSON/Markdown references

Every synthetic sample must carry:

  • language
  • script
  • Unicode ranges observed
  • font
  • source text hash
  • rendered image hash
  • template id
  • degradation id
  • train/eval split id

Handwriting Synthesis

Handwriting is not a v1 blocker. It can be a later ablation.

Possible direction:

  • Use available fonts and handwritten-style fonts for cheap diversity.
  • Explore diffusion/DM-based handwriting generation only after print OCR is stable.
  • For Limbu, do not synthesize handwriting until we have real handwritten or community-reviewed glyph examples. Otherwise we risk generating plausible but wrong orthography.

If we explore One-DM or related diffusion methods, it should be framed as optional augmentation, not the main claim.

Validation Gates

Gate 0: Source Audit

Pass only if:

  • Gorkhapatra article image URLs are downloadable.
  • Epaper direct PDFs are downloadable.
  • PDF text/image extraction path is implemented and audited.
  • Every source artifact is hashed.
  • Source license/terms risk is recorded.

Gate 1: Language Registry

Pass only if every target language has:

  • language name
  • ISO code if known
  • script name
  • Unicode range
  • font list
  • corpus source
  • renderability audit
  • normalization policy
  • PaddleOCR dictionary path
  • evaluation availability

Gate 2: Limbu First

Pass only if:

  • Limbu Unicode rendering works with selected fonts.
  • Limbu PaddleOCR dictionary exists.
  • Limbu synthetic crop/page generator emits valid labels.
  • Limbu held-out real or manually reviewed eval exists.
  • Stock baselines are recorded before fine-tuning.
  • Claim-grade recognizer bundles use --require-eval-real; rendered synthetic Limbu rows may train/ablate but must not appear in held-out eval evidence.
  • Gorkhapatra language-page candidates are promoted through ocrtech finalize-gorkhapatra-language-page-review only after accepted review rows have verified structured reference JSON with text, reading order, tables, and figures.
  • ocrtech audit-gorkhapatra-language-page-review must pass before finalization: exactly one page candidate is accepted per publication/language/date group, and accepted rows point to reference files. Use --require-verified-references for the final pre-claim gate.
  • ocrtech prepare-gorkhapatra-language-page-reference-templates creates draft structured references for reviewers without making them claim-eligible.

Gate 3: PaddleOCR Baselines

Run before any claim:

  • stock PP-OCRv6
  • stock PP-OCRv5 if available
  • Tesseract where traineddata exists
  • Surya
  • GLM-OCR
  • PaddleOCR-VL-1.6

Gate 4: Fine-Tuned Candidate

The candidate must beat baselines on:

  • CER/WER per language
  • strict and normalized Unicode CER
  • exact-line accuracy
  • reading order
  • table text and structure
  • figure/caption detection
  • Markdown reconstruction
  • JSON artifact completeness

English and Nepali must not regress versus the chosen stock baseline.

Gate 5: Claim Review

The paper claim is allowed only if:

  • all public/real validation rows have complete artifacts
  • fallback usage is reported separately
  • synthetic-only improvements are not used as SOTA evidence
  • failures are included in error analysis
  • a reproducibility bundle can rerun the benchmark

Immediate Implementation Steps

  1. Add ocrtech audit-gorkhapatra-source. Implemented: the command audits Naya Nepal article images and Gorkhapatra epaper PDF links, resolves PDF.js viewer URLs to direct /uploads/file/...pdf URLs, optionally downloads and hashes assets, and records PyMuPDF text-layer metadata when available.

  2. Add ocrtech prepare-gorkhapatra-pack. Implemented: the command turns a source-audit JSON file into a pending-reference validation manifest, copies audited article images, renders audited epaper PDF pages when PyMuPDF is available, writes placeholder reference JSON files, and marks samples as reference_status=pending_manual_label and claim_evidence_eligible=false until references are verified.

  3. Add languages/nepal-v1.json registry with Limbu first and at least ten target languages. Implemented: the registry tracks Limbu/Sirijonga first, more than ten Nepal-language targets, English as a non-counted regression language, script/Unicode ranges, font candidates, corpus sources, Gorkhapatra labels, validation status, and PaddleOCR dictionary targets. The ocrtech audit-language-registry command validates the registry and writes JSON/Markdown audit artifacts.

  4. Add renderability audit for Unicode ranges and fonts. Implemented: the ocrtech audit-font-renderability command reads TTF/OTF/TTC cmap tables, checks registered font candidates against each script's Unicode ranges, and writes JSON/Markdown evidence. Raster sample rendering is supported when Pillow is installed. Current cmap evidence verifies Limbu, Devanagari, Santhali/Ol Chiki, and Latin coverage when the registered runtime font assets are present.

  5. Add target-language readiness audit. Implemented: the ocrtech audit-language-readiness command combines languages/nepal-v1.json with passed synthesis-text-manifest-audit.json evidence and writes a per-language checklist. Current run:

    uv run ocrtech audit-language-readiness \
      languages/nepal-v1.json \
      --out outputs/language-readiness-audit-2026-06-16 \
      --synthesis-text-audit outputs/limbu-unicode-text-smoke-audit-2026-06-16/synthesis-text-manifest-audit.json \
      --synthesis-text-audit outputs/tamang-text-smoke-audit-2026-06-16/synthesis-text-manifest-audit.json \
      --synthesis-text-audit outputs/magar-text-smoke-audit-2026-06-16/synthesis-text-manifest-audit.json \
      --synthesis-text-audit outputs/remote-bible-brain-readiness-2026-06-16/synthesis-text-manifest-audit.json \
      --synthesis-text-audit outputs/remote-bible-brain-readiness-extra-2026-06-16/synthesis-text-manifest-audit.json
    

    Result: passed=True target_languages=17 synthesis_ready=11/1. Limbu is synthesis-text ready with 200 audited samples, Tamang is synthesis-text ready with 300 audited Devanagari samples, and Magar is synthesis-text ready with 200 audited lexicon-derived samples. Remote Bible Brain adds 80 audited Devanagari samples each for Nepali, Rana Tharu, and Kulung, plus 60 audited samples each for Gurung, Dhimal, Sunuwar, Rajbanshi, and Sherpa. Every other target language is still a source-candidate or registry-only item until a converter and audit-synthesis-text-manifest run exists.

  6. Add Limbu synthetic line/page generator.

  7. Add PP-OCRv6 stock baseline runner.

  8. Build the first validation pack through Gorkhapatra/Naya Nepal publication date alignment: extract Naya Nepal card dates, match the same-date Gorkhapatra epaper PDF, locate the full भाषा पृष्ठ page, render that PDF page, and keep it pending manual reference labeling. See Gorkhapatra Language Page Alignment.

  9. Keep synthesis resource audits current on cdjk@100.110.148.43, especially the 4TB SSD paths /Volumes/TRANSCEND and ~/github/llm, before creating real-text synthesis manifests.

  10. Use audit-synthesis-resources output as the inventory gate; then build schema-specific converters for Limbu, Tamang, Magar, and remote Bible Brain resources only after actual fields are verified.

External References