Download ocr-tech/docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 14.2 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md
-
curl -L -o MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md
Multilingual PaddleOCR Research Direction
This note records the pivot from a Nepali/English recognizer experiment to a PaddleOCR-centered low-resource document OCR program for Nepal's languages. The research claim should be edited before more GPU time is spent: we are no longer trying to beat the world with a tiny Nepali line dataset. The credible angle is script-aware, low-resource adaptation of PaddleOCR for multilingual Nepali documents, with document-structure validation strong enough to reject weak claims.
Current Decision
PaddleOCR should be the primary engine family.
The reason is practical, not ideological:
- PP-OCRv6 is now the strongest small/local PaddleOCR text OCR direction, with official tiny/small/medium tiers and a unified multilingual recognizer for Chinese, English, Japanese, and Latin-script languages.
- PP-StructureV3 is the right local document-parsing scaffold for layout, tables, reading order, formulas, seals, and document reconstruction.
- PaddleOCR-VL-1.6 is the right fallback/teacher for hard pages, charts, figures, seals, and pages where a small local pipeline has low confidence.
- PaddleOCR already has the training/deployment stack we need: dictionaries, recognition configs, layout modules, table modules, model export, and CPU/GPU deployment paths.
The caveat is important: PP-OCRv6's headline multilingual coverage does not solve Limbu, Kirat Rai, Tamang Tamyig, or other Nepal-specific scripts by default. We should use PaddleOCR as the framework and base model family, then train or adapt script-specific recognition heads/dictionaries where needed.
Revised Research Thesis
Candidate thesis:
We build a PaddleOCR-based multilingual document OCR engine for Nepali, English, and low-resource languages of Nepal by combining script-aware corpus construction, Unicode-safe synthetic document generation, PaddleOCR recognizer fine-tuning, PP-Structure-style document parsing, and selective VLM fallback. We evaluate against Tesseract, stock PaddleOCR/PP-OCR, Surya, GLM-OCR, and PaddleOCR-VL on OCR accuracy, reading order, table structure, figure/caption handling, and Markdown/JSON reconstruction.
This is stronger than "Nepali OCR beats Tesseract" because it contributes:
- a multilingual/script inventory for Nepal document OCR
- a reproducible synthetic document generator grounded in real text resources
- a PaddleOCR adaptation recipe for low-resource scripts/languages
- held-out public plus real-document validation rather than synthetic-only claims
- a clear local engine plus optional VLM fallback design
Language And Script Inventory
OCR support should be tracked by (language, script, Unicode, font, corpus, evaluation) rather than by language name alone.
| Target | Likely script(s) | Unicode status | Current local resources found | OCR implication |
|---|---|---|---|---|
| Nepali | Devanagari | U+0900..U+097F plus extensions as needed | Existing OCR manifests, G2P/text frontend, Loksewa/Gorkhapatra-like documents | Keep as core training/eval language |
| English | Latin | Basic/extended Latin | NVIDIA synthetic line crops, document text | Preserve English, do not regress |
| Tamang | Mostly Devanagari in local corpora; Tamyig/Tibetan-related script also exists | Devanagari is supported; Tamyig needs separate font/encoding audit | ~/github/llm/ltk/tamang: NepTam train/test, NepTam80K synthetic, dictionary, TTS lexicon |
Start with Devanagari Tamang; audit Tamyig separately |
| Limbu/Yakthung | Limbu/Sirijonga | Limbu block U+1900..U+194F | ~/github/llm/ltk/limbu: Limbu dictionary, Pangloss IGT, lexicon, G2P |
Needs Limbu fonts, Paddle dictionary, synthetic rendering, held-out real pages |
| Rai/Bantawa/Kirat Rai | Kirat Rai and Devanagari/Latin transcriptions depending source | Kirat Rai block U+16D40..U+16D7F in Unicode 16.0 | No local corpus confirmed yet; web sources point to Sikkim Herald/Rai textbooks/proposal examples | Needs corpus acquisition and fonts before training |
| Magar | Primarily Devanagari; Magar Akkha is a community-script issue | Devanagari supported; Magar Akkha status is conflicting in web sources and must be verified against the current Unicode database before use | ~/github/llm/ltk/magar: Western Magar dictionary and resource report |
Start with Devanagari Magar; treat Akkha as research/audit item |
| Nepal Bhasa/Newar | Newar/Prachalit, Bhujimol, Ranjana, and Devanagari | Prachalit uses Newa U+11400..U+1147F; Bhujimol has no separate script encoding and uses an explicit Newa-transcription style profile; Ranjana remains unencoded; Devanagari is separate | Four routes are structurally audited; Prachalit and Devanagari have diagnostic synthetic runtime smokes, Bhujimol and Ranjana have exact reviewer references and acquired review candidates, and one permission-covered official Newa edition is cataloged for remote script screening | Keep four visual profiles separate; references and acquisition candidates are not recognizer-quality evidence, and shared transcription codepoints do not imply visual-model coverage |
Do not call these "dialects" in the paper unless the linguistic unit really is a dialect. OCR sees scripts and glyph distributions first. Tamang varieties, Rai languages, Magar varieties, and Limbu are not one generic dialect bucket.
Local Resource Inventory
Verified local files on this machine:
Tamang
Directory: ~/github/llm/ltk/tamang
data/neptam_train.csv: 15,003 lines including header.data/neptam_test.csv: 5,003 lines including header.data/neptam80k_synthetic.csv: 80,100 lines including header.data/western_tamang_dictionary.tsv: 3,605 lines including header.data/tts_lexicon.tsv: 107,225 lines.
Observed fields:
neptam_train.csv:sentence_id,nepali_sentences,translation_tamang,sentence_type,Tense,polaritywestern_tamang_dictionary.tsv:headword,pos,definition
This is enough to synthesize Devanagari Tamang/Nepali line crops and document pages immediately, subject to license review.
Limbu
Directory: ~/github/llm/ltk/limbu
data/limdic_dictionary.tsv: 8,798 lines including header.data/pangloss_dictionary.tsv: 2,541 lines including header.data/pangloss_igt_corpus.tsv: 10,951 lines including header.data/pangloss_nepali_loanwords.tsv: 353 lines including header.data/tts_lexicon.tsv: 2,541 lines including header.g2p/limbu_g2p.py: Unicode Limbu to phoneme tooling.
Observed fields:
limdic_dictionary.tsv:id,headword_limbu,desc_devanagari,group,definition,ipapangloss_dictionary.tsv:headword,pos,definitions,nepali,etymology,root,example_count
This is enough for first-pass Limbu synthetic line generation, but it needs font and rendering validation before recognizer training.
Magar
Directory: ~/github/llm/ltk/magar
data/western_magar_dictionary.tsv: 583 lines including header.data/magar_resource_report.md: source survey.
Observed fields:
western_magar_dictionary.tsv:headword,ipa,pos,variant,definition
This is too small for a standalone Magar OCR claim, but useful as seed text for synthetic rendering and dictionary coverage tests.
Other Nepal-Focused Sources
Potentially useful local directories:
~/github/llm/loksewa: many Nepali PDFs, PNGs, and Markdown documents with government-exam style layouts.~/github/llm/g2pand~/github/llm/nepali_text_frontend: Nepali text normalization, lexicon, and akshara resources.~/github/llm/new-place: Nepal-focused scraped text archives.
The requested remote host cdjk@100.110.148.43 is reachable, but this machine
does not have sshpass. An expect attempt reached the password prompt but
did not complete cleanly in this run. Remote corpus inventory remains pending.
Data Synthesis Strategy
Synthesize documents, not just isolated words.
1. Text Layer
Use real text wherever possible:
- Nepali: local G2P/frontend corpora, Loksewa/government-style pages, public domain/news/government sources after license audit.
- Tamang: NepTam train/test and dictionary material.
- Limbu: Unicode Limbu dictionary entries, Pangloss-derived text, loanwords, and IGT corpus rows.
- Magar: Western Magar dictionary entries and later Webonary/app-derived data only if legally extractable.
- Rai/Bantawa/Kirat Rai: Gorkhapatra/Naya Nepal, Sikkim Herald/Rai materials, textbooks, and Unicode proposal examples only after acquisition and license review.
2. Script And Font Layer
For every target script, create a renderability audit:
- Unicode block coverage.
- Fonts available on macOS/Linux/AWS.
- Shaping correctness with HarfBuzz/Pango.
- Unsupported character list.
- Confusable/normalization policy.
- PaddleOCR dictionary file.
- Grapheme/akshara segmentation tests.
No sample should enter training unless its text can be rendered and normalized deterministically.
3. OCR Crop Generation
Generate:
- word crops
- line crops
- table-cell crops
- caption crops
- noisy crops extracted from rendered pages
For PaddleOCR training, each generated crop needs:
- image path
- exact UTF-8 label
- script/language metadata
- source corpus id
- source text hash
- render template id
- font id
- degradation id
4. Full Document Generation
Render complete pages with ground truth:
- single-column and two-column articles
- school notices
- government forms
- invoices/receipts
- Naya Nepal/Gorkhapatra-style multilingual pages
- tables with merged cells
- figure/chart blocks with captions
- headers/footers/page numbers
For every page emit:
document.jsonreference with blocks, lines, reading order, tables, figures, captions, and bboxesdocument.mdreference- table CSV/HTML references
- crop manifests for recognizer training
5. Degradation Layer
Apply realistic scan/photo transforms:
- blur
- skew
- rotation
- low contrast
- JPEG compression
- bleed-through/noise
- perspective warp
- uneven lighting
- cropped margins
- phone-photo shadows
The degradation must be recorded in metadata so evaluation can report which conditions fail.
PaddleOCR Training Plan
Recommended branches:
PP-OCRv6 baseline audit
- Run stock PP-OCRv6 on our current Nepali/English eval packs.
- Add Tamang/Limbu smoke pages after fonts render correctly.
- Record failures by script and degradation.
Devanagari multilingual recognizer
- Fine-tune PaddleOCR recognizer for Nepali + English + Devanagari Tamang + Devanagari Magar/Rai where text exists.
- Dictionary: Devanagari, Latin, punctuation, Devanagari digits, Arabic digits, symbols used in forms/tables.
- Gate: must beat stock PP-OCRv6/PP-OCRv5 and Tesseract on the same Devanagari multilingual slices without English regression.
Limbu recognizer
- Build a separate Limbu/Sirijonga dictionary and synthetic corpus.
- Start with line/cell OCR, then document pages.
- Gate: beat stock PaddleOCR fallback behavior and any available Tesseract baseline if traineddata exists; otherwise compare to PaddleOCR-VL and report as new low-resource benchmark rather than broad SOTA.
Kirat Rai recognizer
- Do not train until corpus/font audit is complete.
- Unicode 16.0 support means runtime/font stacks may lag; verify first.
PP-Structure integration
- Keep the document pipeline engine-side: layout, reading order, table reconstruction, figure/caption handling, Markdown/JSON export.
- Replace recognizer component per script/domain.
- Use PaddleOCR-VL only as selective fallback or teacher, not as the default "small local" engine.
Validation Plan
The revised claim needs these evaluation groups:
Recognizer OCR
- CER/WER per language.
- CER/WER per script.
- Exact-line accuracy.
- Unicode-normalized and strict metrics.
- English non-regression.
Document Understanding
- Reading order.
- Markdown reconstruction.
- Table cell text and structure.
- HTML/TEDS-style table similarity.
- Figure/caption detection.
- Required output artifact checks.
Baselines
- Tesseract where language/script data exists.
- Stock PaddleOCR PP-OCRv5/PP-OCRv6.
- Surya.
- GLM-OCR.
- PaddleOCR-VL-1.6.
Splits
- Synthetic train.
- Synthetic held-out by source text/template/font/degradation.
- Real held-out documents.
- Public benchmarks separate from all training.
No SOTA claim should be accepted from synthetic-only evaluation. Synthetic data is for scale and controlled ablations. Real and public held-out data are for claims.
Immediate Engineering Tasks
- Add a
language_registryfile describing language, script, Unicode ranges, required fonts, normalization policy, and Paddle dictionary path. - Add corpus importers for the verified local Tamang, Limbu, and Magar files.
- Add a renderability audit command that fails on unsupported characters, missing glyphs, bad shaping, or normalization drift.
- Add a synthetic line/page generator that emits PaddleOCR-compatible labels plus document references.
- Run a stock PP-OCRv6 baseline audit before the next fine-tune.
- Only then start CUDA fine-tuning.
Source Pointers
- PaddleOCR docs index, including PP-OCRv6, PP-StructureV3, and PaddleOCR-VL: https://paddlepaddle.github.io/PaddleOCR/main/en/index.html
- PaddleOCR repository: https://github.com/PaddlePaddle/PaddleOCR
- PaddleOCR PP-StructureV3 pipeline docs: https://paddlepaddle.github.io/PaddleOCR/main/en/version3.x/pipeline_usage/PP-StructureV3.html
- PaddleOCR-VL-1.6 docs: https://paddlepaddle.github.io/PaddleOCR/main/en/version3.x/algorithm/PaddleOCR-VL/PaddleOCR-VL-1.6.html
- Unicode Limbu chart: https://www.unicode.org/charts/PDF/U1900.pdf
- Unicode Kirat Rai proposal: https://www.unicode.org/L2/L2022/22043-kirat-rai.pdf
- Kirat Rai orthography notes: https://r12a.github.io/scripts/krai/bap
- Unicode Devanagari core-spec chapter: https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-12/
- Magar Akkha proposal: https://www.unicode.org/L2/L2011/11144-magar-akkha.pdf