# Multilingual PaddleOCR Research Direction This note records the pivot from a Nepali/English recognizer experiment to a PaddleOCR-centered low-resource document OCR program for Nepal's languages. The research claim should be edited before more GPU time is spent: we are no longer trying to beat the world with a tiny Nepali line dataset. The credible angle is script-aware, low-resource adaptation of PaddleOCR for multilingual Nepali documents, with document-structure validation strong enough to reject weak claims. ## Current Decision PaddleOCR should be the primary engine family. The reason is practical, not ideological: - PP-OCRv6 is now the strongest small/local PaddleOCR text OCR direction, with official tiny/small/medium tiers and a unified multilingual recognizer for Chinese, English, Japanese, and Latin-script languages. - PP-StructureV3 is the right local document-parsing scaffold for layout, tables, reading order, formulas, seals, and document reconstruction. - PaddleOCR-VL-1.6 is the right fallback/teacher for hard pages, charts, figures, seals, and pages where a small local pipeline has low confidence. - PaddleOCR already has the training/deployment stack we need: dictionaries, recognition configs, layout modules, table modules, model export, and CPU/GPU deployment paths. The caveat is important: PP-OCRv6's headline multilingual coverage does not solve Limbu, Kirat Rai, Tamang Tamyig, or other Nepal-specific scripts by default. We should use PaddleOCR as the framework and base model family, then train or adapt script-specific recognition heads/dictionaries where needed. ## Revised Research Thesis Candidate thesis: > We build a PaddleOCR-based multilingual document OCR engine for Nepali, > English, and low-resource languages of Nepal by combining script-aware corpus > construction, Unicode-safe synthetic document generation, PaddleOCR > recognizer fine-tuning, PP-Structure-style document parsing, and selective > VLM fallback. We evaluate against Tesseract, stock PaddleOCR/PP-OCR, Surya, > GLM-OCR, and PaddleOCR-VL on OCR accuracy, reading order, table structure, > figure/caption handling, and Markdown/JSON reconstruction. This is stronger than "Nepali OCR beats Tesseract" because it contributes: - a multilingual/script inventory for Nepal document OCR - a reproducible synthetic document generator grounded in real text resources - a PaddleOCR adaptation recipe for low-resource scripts/languages - held-out public plus real-document validation rather than synthetic-only claims - a clear local engine plus optional VLM fallback design ## Language And Script Inventory OCR support should be tracked by `(language, script, Unicode, font, corpus, evaluation)` rather than by language name alone. | Target | Likely script(s) | Unicode status | Current local resources found | OCR implication | | --- | --- | --- | --- | --- | | Nepali | Devanagari | U+0900..U+097F plus extensions as needed | Existing OCR manifests, G2P/text frontend, Loksewa/Gorkhapatra-like documents | Keep as core training/eval language | | English | Latin | Basic/extended Latin | NVIDIA synthetic line crops, document text | Preserve English, do not regress | | Tamang | Mostly Devanagari in local corpora; Tamyig/Tibetan-related script also exists | Devanagari is supported; Tamyig needs separate font/encoding audit | `~/github/llm/ltk/tamang`: NepTam train/test, NepTam80K synthetic, dictionary, TTS lexicon | Start with Devanagari Tamang; audit Tamyig separately | | Limbu/Yakthung | Limbu/Sirijonga | Limbu block U+1900..U+194F | `~/github/llm/ltk/limbu`: Limbu dictionary, Pangloss IGT, lexicon, G2P | Needs Limbu fonts, Paddle dictionary, synthetic rendering, held-out real pages | | Rai/Bantawa/Kirat Rai | Kirat Rai and Devanagari/Latin transcriptions depending source | Kirat Rai block U+16D40..U+16D7F in Unicode 16.0 | No local corpus confirmed yet; web sources point to Sikkim Herald/Rai textbooks/proposal examples | Needs corpus acquisition and fonts before training | | Magar | Primarily Devanagari; Magar Akkha is a community-script issue | Devanagari supported; Magar Akkha status is conflicting in web sources and must be verified against the current Unicode database before use | `~/github/llm/ltk/magar`: Western Magar dictionary and resource report | Start with Devanagari Magar; treat Akkha as research/audit item | | Nepal Bhasa/Newar | Newar/Prachalit, Bhujimol, Ranjana, and Devanagari | Prachalit uses Newa U+11400..U+1147F; Bhujimol has no separate script encoding and uses an explicit Newa-transcription style profile; Ranjana remains unencoded; Devanagari is separate | Four routes are structurally audited; Prachalit and Devanagari have diagnostic synthetic runtime smokes, Bhujimol and Ranjana have exact reviewer references and acquired review candidates, and one permission-covered official Newa edition is cataloged for remote script screening | Keep four visual profiles separate; references and acquisition candidates are not recognizer-quality evidence, and shared transcription codepoints do not imply visual-model coverage | Do not call these "dialects" in the paper unless the linguistic unit really is a dialect. OCR sees scripts and glyph distributions first. Tamang varieties, Rai languages, Magar varieties, and Limbu are not one generic dialect bucket. ## Local Resource Inventory Verified local files on this machine: ### Tamang Directory: `~/github/llm/ltk/tamang` - `data/neptam_train.csv`: 15,003 lines including header. - `data/neptam_test.csv`: 5,003 lines including header. - `data/neptam80k_synthetic.csv`: 80,100 lines including header. - `data/western_tamang_dictionary.tsv`: 3,605 lines including header. - `data/tts_lexicon.tsv`: 107,225 lines. Observed fields: - `neptam_train.csv`: `sentence_id,nepali_sentences,translation_tamang,sentence_type,Tense,polarity` - `western_tamang_dictionary.tsv`: `headword,pos,definition` This is enough to synthesize Devanagari Tamang/Nepali line crops and document pages immediately, subject to license review. ### Limbu Directory: `~/github/llm/ltk/limbu` - `data/limdic_dictionary.tsv`: 8,798 lines including header. - `data/pangloss_dictionary.tsv`: 2,541 lines including header. - `data/pangloss_igt_corpus.tsv`: 10,951 lines including header. - `data/pangloss_nepali_loanwords.tsv`: 353 lines including header. - `data/tts_lexicon.tsv`: 2,541 lines including header. - `g2p/limbu_g2p.py`: Unicode Limbu to phoneme tooling. Observed fields: - `limdic_dictionary.tsv`: `id,headword_limbu,desc_devanagari,group,definition,ipa` - `pangloss_dictionary.tsv`: `headword,pos,definitions,nepali,etymology,root,example_count` This is enough for first-pass Limbu synthetic line generation, but it needs font and rendering validation before recognizer training. ### Magar Directory: `~/github/llm/ltk/magar` - `data/western_magar_dictionary.tsv`: 583 lines including header. - `data/magar_resource_report.md`: source survey. Observed fields: - `western_magar_dictionary.tsv`: `headword,ipa,pos,variant,definition` This is too small for a standalone Magar OCR claim, but useful as seed text for synthetic rendering and dictionary coverage tests. ### Other Nepal-Focused Sources Potentially useful local directories: - `~/github/llm/loksewa`: many Nepali PDFs, PNGs, and Markdown documents with government-exam style layouts. - `~/github/llm/g2p` and `~/github/llm/nepali_text_frontend`: Nepali text normalization, lexicon, and akshara resources. - `~/github/llm/new-place`: Nepal-focused scraped text archives. The requested remote host `cdjk@100.110.148.43` is reachable, but this machine does not have `sshpass`. An `expect` attempt reached the password prompt but did not complete cleanly in this run. Remote corpus inventory remains pending. ## Data Synthesis Strategy Synthesize documents, not just isolated words. ### 1. Text Layer Use real text wherever possible: - Nepali: local G2P/frontend corpora, Loksewa/government-style pages, public domain/news/government sources after license audit. - Tamang: NepTam train/test and dictionary material. - Limbu: Unicode Limbu dictionary entries, Pangloss-derived text, loanwords, and IGT corpus rows. - Magar: Western Magar dictionary entries and later Webonary/app-derived data only if legally extractable. - Rai/Bantawa/Kirat Rai: Gorkhapatra/Naya Nepal, Sikkim Herald/Rai materials, textbooks, and Unicode proposal examples only after acquisition and license review. ### 2. Script And Font Layer For every target script, create a renderability audit: - Unicode block coverage. - Fonts available on macOS/Linux/AWS. - Shaping correctness with HarfBuzz/Pango. - Unsupported character list. - Confusable/normalization policy. - PaddleOCR dictionary file. - Grapheme/akshara segmentation tests. No sample should enter training unless its text can be rendered and normalized deterministically. ### 3. OCR Crop Generation Generate: - word crops - line crops - table-cell crops - caption crops - noisy crops extracted from rendered pages For PaddleOCR training, each generated crop needs: - image path - exact UTF-8 label - script/language metadata - source corpus id - source text hash - render template id - font id - degradation id ### 4. Full Document Generation Render complete pages with ground truth: - single-column and two-column articles - school notices - government forms - invoices/receipts - Naya Nepal/Gorkhapatra-style multilingual pages - tables with merged cells - figure/chart blocks with captions - headers/footers/page numbers For every page emit: - `document.json` reference with blocks, lines, reading order, tables, figures, captions, and bboxes - `document.md` reference - table CSV/HTML references - crop manifests for recognizer training ### 5. Degradation Layer Apply realistic scan/photo transforms: - blur - skew - rotation - low contrast - JPEG compression - bleed-through/noise - perspective warp - uneven lighting - cropped margins - phone-photo shadows The degradation must be recorded in metadata so evaluation can report which conditions fail. ## PaddleOCR Training Plan Recommended branches: 1. **PP-OCRv6 baseline audit** - Run stock PP-OCRv6 on our current Nepali/English eval packs. - Add Tamang/Limbu smoke pages after fonts render correctly. - Record failures by script and degradation. 2. **Devanagari multilingual recognizer** - Fine-tune PaddleOCR recognizer for Nepali + English + Devanagari Tamang + Devanagari Magar/Rai where text exists. - Dictionary: Devanagari, Latin, punctuation, Devanagari digits, Arabic digits, symbols used in forms/tables. - Gate: must beat stock PP-OCRv6/PP-OCRv5 and Tesseract on the same Devanagari multilingual slices without English regression. 3. **Limbu recognizer** - Build a separate Limbu/Sirijonga dictionary and synthetic corpus. - Start with line/cell OCR, then document pages. - Gate: beat stock PaddleOCR fallback behavior and any available Tesseract baseline if traineddata exists; otherwise compare to PaddleOCR-VL and report as new low-resource benchmark rather than broad SOTA. 4. **Kirat Rai recognizer** - Do not train until corpus/font audit is complete. - Unicode 16.0 support means runtime/font stacks may lag; verify first. 5. **PP-Structure integration** - Keep the document pipeline engine-side: layout, reading order, table reconstruction, figure/caption handling, Markdown/JSON export. - Replace recognizer component per script/domain. - Use PaddleOCR-VL only as selective fallback or teacher, not as the default "small local" engine. ## Validation Plan The revised claim needs these evaluation groups: 1. **Recognizer OCR** - CER/WER per language. - CER/WER per script. - Exact-line accuracy. - Unicode-normalized and strict metrics. - English non-regression. 2. **Document Understanding** - Reading order. - Markdown reconstruction. - Table cell text and structure. - HTML/TEDS-style table similarity. - Figure/caption detection. - Required output artifact checks. 3. **Baselines** - Tesseract where language/script data exists. - Stock PaddleOCR PP-OCRv5/PP-OCRv6. - Surya. - GLM-OCR. - PaddleOCR-VL-1.6. 4. **Splits** - Synthetic train. - Synthetic held-out by source text/template/font/degradation. - Real held-out documents. - Public benchmarks separate from all training. No SOTA claim should be accepted from synthetic-only evaluation. Synthetic data is for scale and controlled ablations. Real and public held-out data are for claims. ## Immediate Engineering Tasks 1. Add a `language_registry` file describing language, script, Unicode ranges, required fonts, normalization policy, and Paddle dictionary path. 2. Add corpus importers for the verified local Tamang, Limbu, and Magar files. 3. Add a renderability audit command that fails on unsupported characters, missing glyphs, bad shaping, or normalization drift. 4. Add a synthetic line/page generator that emits PaddleOCR-compatible labels plus document references. 5. Run a stock PP-OCRv6 baseline audit before the next fine-tune. 6. Only then start CUDA fine-tuning. ## Source Pointers - PaddleOCR docs index, including PP-OCRv6, PP-StructureV3, and PaddleOCR-VL: - PaddleOCR repository: - PaddleOCR PP-StructureV3 pipeline docs: - PaddleOCR-VL-1.6 docs: - Unicode Limbu chart: - Unicode Kirat Rai proposal: - Kirat Rai orthography notes: - Unicode Devanagari core-spec chapter: - Magar Akkha proposal: