ocr-projects-archive / ocr-tech /docs /MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
14.2 kB
# Multilingual PaddleOCR Research Direction
This note records the pivot from a Nepali/English recognizer experiment to a
PaddleOCR-centered low-resource document OCR program for Nepal's languages. The
research claim should be edited before more GPU time is spent: we are no longer
trying to beat the world with a tiny Nepali line dataset. The credible angle is
script-aware, low-resource adaptation of PaddleOCR for multilingual Nepali
documents, with document-structure validation strong enough to reject weak
claims.
## Current Decision
PaddleOCR should be the primary engine family.
The reason is practical, not ideological:
- PP-OCRv6 is now the strongest small/local PaddleOCR text OCR direction, with
official tiny/small/medium tiers and a unified multilingual recognizer for
Chinese, English, Japanese, and Latin-script languages.
- PP-StructureV3 is the right local document-parsing scaffold for layout,
tables, reading order, formulas, seals, and document reconstruction.
- PaddleOCR-VL-1.6 is the right fallback/teacher for hard pages, charts,
figures, seals, and pages where a small local pipeline has low confidence.
- PaddleOCR already has the training/deployment stack we need: dictionaries,
recognition configs, layout modules, table modules, model export, and CPU/GPU
deployment paths.
The caveat is important: PP-OCRv6's headline multilingual coverage does not
solve Limbu, Kirat Rai, Tamang Tamyig, or other Nepal-specific scripts by
default. We should use PaddleOCR as the framework and base model family, then
train or adapt script-specific recognition heads/dictionaries where needed.
## Revised Research Thesis
Candidate thesis:
> We build a PaddleOCR-based multilingual document OCR engine for Nepali,
> English, and low-resource languages of Nepal by combining script-aware corpus
> construction, Unicode-safe synthetic document generation, PaddleOCR
> recognizer fine-tuning, PP-Structure-style document parsing, and selective
> VLM fallback. We evaluate against Tesseract, stock PaddleOCR/PP-OCR, Surya,
> GLM-OCR, and PaddleOCR-VL on OCR accuracy, reading order, table structure,
> figure/caption handling, and Markdown/JSON reconstruction.
This is stronger than "Nepali OCR beats Tesseract" because it contributes:
- a multilingual/script inventory for Nepal document OCR
- a reproducible synthetic document generator grounded in real text resources
- a PaddleOCR adaptation recipe for low-resource scripts/languages
- held-out public plus real-document validation rather than synthetic-only
claims
- a clear local engine plus optional VLM fallback design
## Language And Script Inventory
OCR support should be tracked by `(language, script, Unicode, font, corpus,
evaluation)` rather than by language name alone.
| Target | Likely script(s) | Unicode status | Current local resources found | OCR implication |
| --- | --- | --- | --- | --- |
| Nepali | Devanagari | U+0900..U+097F plus extensions as needed | Existing OCR manifests, G2P/text frontend, Loksewa/Gorkhapatra-like documents | Keep as core training/eval language |
| English | Latin | Basic/extended Latin | NVIDIA synthetic line crops, document text | Preserve English, do not regress |
| Tamang | Mostly Devanagari in local corpora; Tamyig/Tibetan-related script also exists | Devanagari is supported; Tamyig needs separate font/encoding audit | `~/github/llm/ltk/tamang`: NepTam train/test, NepTam80K synthetic, dictionary, TTS lexicon | Start with Devanagari Tamang; audit Tamyig separately |
| Limbu/Yakthung | Limbu/Sirijonga | Limbu block U+1900..U+194F | `~/github/llm/ltk/limbu`: Limbu dictionary, Pangloss IGT, lexicon, G2P | Needs Limbu fonts, Paddle dictionary, synthetic rendering, held-out real pages |
| Rai/Bantawa/Kirat Rai | Kirat Rai and Devanagari/Latin transcriptions depending source | Kirat Rai block U+16D40..U+16D7F in Unicode 16.0 | No local corpus confirmed yet; web sources point to Sikkim Herald/Rai textbooks/proposal examples | Needs corpus acquisition and fonts before training |
| Magar | Primarily Devanagari; Magar Akkha is a community-script issue | Devanagari supported; Magar Akkha status is conflicting in web sources and must be verified against the current Unicode database before use | `~/github/llm/ltk/magar`: Western Magar dictionary and resource report | Start with Devanagari Magar; treat Akkha as research/audit item |
| Nepal Bhasa/Newar | Newar/Prachalit, Bhujimol, Ranjana, and Devanagari | Prachalit uses Newa U+11400..U+1147F; Bhujimol has no separate script encoding and uses an explicit Newa-transcription style profile; Ranjana remains unencoded; Devanagari is separate | Four routes are structurally audited; Prachalit and Devanagari have diagnostic synthetic runtime smokes, Bhujimol and Ranjana have exact reviewer references and acquired review candidates, and one permission-covered official Newa edition is cataloged for remote script screening | Keep four visual profiles separate; references and acquisition candidates are not recognizer-quality evidence, and shared transcription codepoints do not imply visual-model coverage |
Do not call these "dialects" in the paper unless the linguistic unit really is
a dialect. OCR sees scripts and glyph distributions first. Tamang varieties,
Rai languages, Magar varieties, and Limbu are not one generic dialect bucket.
## Local Resource Inventory
Verified local files on this machine:
### Tamang
Directory: `~/github/llm/ltk/tamang`
- `data/neptam_train.csv`: 15,003 lines including header.
- `data/neptam_test.csv`: 5,003 lines including header.
- `data/neptam80k_synthetic.csv`: 80,100 lines including header.
- `data/western_tamang_dictionary.tsv`: 3,605 lines including header.
- `data/tts_lexicon.tsv`: 107,225 lines.
Observed fields:
- `neptam_train.csv`: `sentence_id,nepali_sentences,translation_tamang,sentence_type,Tense,polarity`
- `western_tamang_dictionary.tsv`: `headword,pos,definition`
This is enough to synthesize Devanagari Tamang/Nepali line crops and document
pages immediately, subject to license review.
### Limbu
Directory: `~/github/llm/ltk/limbu`
- `data/limdic_dictionary.tsv`: 8,798 lines including header.
- `data/pangloss_dictionary.tsv`: 2,541 lines including header.
- `data/pangloss_igt_corpus.tsv`: 10,951 lines including header.
- `data/pangloss_nepali_loanwords.tsv`: 353 lines including header.
- `data/tts_lexicon.tsv`: 2,541 lines including header.
- `g2p/limbu_g2p.py`: Unicode Limbu to phoneme tooling.
Observed fields:
- `limdic_dictionary.tsv`: `id,headword_limbu,desc_devanagari,group,definition,ipa`
- `pangloss_dictionary.tsv`: `headword,pos,definitions,nepali,etymology,root,example_count`
This is enough for first-pass Limbu synthetic line generation, but it needs
font and rendering validation before recognizer training.
### Magar
Directory: `~/github/llm/ltk/magar`
- `data/western_magar_dictionary.tsv`: 583 lines including header.
- `data/magar_resource_report.md`: source survey.
Observed fields:
- `western_magar_dictionary.tsv`: `headword,ipa,pos,variant,definition`
This is too small for a standalone Magar OCR claim, but useful as seed text for
synthetic rendering and dictionary coverage tests.
### Other Nepal-Focused Sources
Potentially useful local directories:
- `~/github/llm/loksewa`: many Nepali PDFs, PNGs, and Markdown documents with
government-exam style layouts.
- `~/github/llm/g2p` and `~/github/llm/nepali_text_frontend`: Nepali text
normalization, lexicon, and akshara resources.
- `~/github/llm/new-place`: Nepal-focused scraped text archives.
The requested remote host `cdjk@100.110.148.43` is reachable, but this machine
does not have `sshpass`. An `expect` attempt reached the password prompt but
did not complete cleanly in this run. Remote corpus inventory remains pending.
## Data Synthesis Strategy
Synthesize documents, not just isolated words.
### 1. Text Layer
Use real text wherever possible:
- Nepali: local G2P/frontend corpora, Loksewa/government-style pages, public
domain/news/government sources after license audit.
- Tamang: NepTam train/test and dictionary material.
- Limbu: Unicode Limbu dictionary entries, Pangloss-derived text, loanwords,
and IGT corpus rows.
- Magar: Western Magar dictionary entries and later Webonary/app-derived data
only if legally extractable.
- Rai/Bantawa/Kirat Rai: Gorkhapatra/Naya Nepal, Sikkim Herald/Rai materials,
textbooks, and Unicode proposal examples only after acquisition and license
review.
### 2. Script And Font Layer
For every target script, create a renderability audit:
- Unicode block coverage.
- Fonts available on macOS/Linux/AWS.
- Shaping correctness with HarfBuzz/Pango.
- Unsupported character list.
- Confusable/normalization policy.
- PaddleOCR dictionary file.
- Grapheme/akshara segmentation tests.
No sample should enter training unless its text can be rendered and normalized
deterministically.
### 3. OCR Crop Generation
Generate:
- word crops
- line crops
- table-cell crops
- caption crops
- noisy crops extracted from rendered pages
For PaddleOCR training, each generated crop needs:
- image path
- exact UTF-8 label
- script/language metadata
- source corpus id
- source text hash
- render template id
- font id
- degradation id
### 4. Full Document Generation
Render complete pages with ground truth:
- single-column and two-column articles
- school notices
- government forms
- invoices/receipts
- Naya Nepal/Gorkhapatra-style multilingual pages
- tables with merged cells
- figure/chart blocks with captions
- headers/footers/page numbers
For every page emit:
- `document.json` reference with blocks, lines, reading order, tables, figures,
captions, and bboxes
- `document.md` reference
- table CSV/HTML references
- crop manifests for recognizer training
### 5. Degradation Layer
Apply realistic scan/photo transforms:
- blur
- skew
- rotation
- low contrast
- JPEG compression
- bleed-through/noise
- perspective warp
- uneven lighting
- cropped margins
- phone-photo shadows
The degradation must be recorded in metadata so evaluation can report which
conditions fail.
## PaddleOCR Training Plan
Recommended branches:
1. **PP-OCRv6 baseline audit**
- Run stock PP-OCRv6 on our current Nepali/English eval packs.
- Add Tamang/Limbu smoke pages after fonts render correctly.
- Record failures by script and degradation.
2. **Devanagari multilingual recognizer**
- Fine-tune PaddleOCR recognizer for Nepali + English + Devanagari Tamang +
Devanagari Magar/Rai where text exists.
- Dictionary: Devanagari, Latin, punctuation, Devanagari digits, Arabic
digits, symbols used in forms/tables.
- Gate: must beat stock PP-OCRv6/PP-OCRv5 and Tesseract on the same
Devanagari multilingual slices without English regression.
3. **Limbu recognizer**
- Build a separate Limbu/Sirijonga dictionary and synthetic corpus.
- Start with line/cell OCR, then document pages.
- Gate: beat stock PaddleOCR fallback behavior and any available Tesseract
baseline if traineddata exists; otherwise compare to PaddleOCR-VL and
report as new low-resource benchmark rather than broad SOTA.
4. **Kirat Rai recognizer**
- Do not train until corpus/font audit is complete.
- Unicode 16.0 support means runtime/font stacks may lag; verify first.
5. **PP-Structure integration**
- Keep the document pipeline engine-side: layout, reading order, table
reconstruction, figure/caption handling, Markdown/JSON export.
- Replace recognizer component per script/domain.
- Use PaddleOCR-VL only as selective fallback or teacher, not as the default
"small local" engine.
## Validation Plan
The revised claim needs these evaluation groups:
1. **Recognizer OCR**
- CER/WER per language.
- CER/WER per script.
- Exact-line accuracy.
- Unicode-normalized and strict metrics.
- English non-regression.
2. **Document Understanding**
- Reading order.
- Markdown reconstruction.
- Table cell text and structure.
- HTML/TEDS-style table similarity.
- Figure/caption detection.
- Required output artifact checks.
3. **Baselines**
- Tesseract where language/script data exists.
- Stock PaddleOCR PP-OCRv5/PP-OCRv6.
- Surya.
- GLM-OCR.
- PaddleOCR-VL-1.6.
4. **Splits**
- Synthetic train.
- Synthetic held-out by source text/template/font/degradation.
- Real held-out documents.
- Public benchmarks separate from all training.
No SOTA claim should be accepted from synthetic-only evaluation. Synthetic data
is for scale and controlled ablations. Real and public held-out data are for
claims.
## Immediate Engineering Tasks
1. Add a `language_registry` file describing language, script, Unicode ranges,
required fonts, normalization policy, and Paddle dictionary path.
2. Add corpus importers for the verified local Tamang, Limbu, and Magar files.
3. Add a renderability audit command that fails on unsupported characters,
missing glyphs, bad shaping, or normalization drift.
4. Add a synthetic line/page generator that emits PaddleOCR-compatible labels
plus document references.
5. Run a stock PP-OCRv6 baseline audit before the next fine-tune.
6. Only then start CUDA fine-tuning.
## Source Pointers
- PaddleOCR docs index, including PP-OCRv6, PP-StructureV3, and PaddleOCR-VL:
<https://paddlepaddle.github.io/PaddleOCR/main/en/index.html>
- PaddleOCR repository: <https://github.com/PaddlePaddle/PaddleOCR>
- PaddleOCR PP-StructureV3 pipeline docs:
<https://paddlepaddle.github.io/PaddleOCR/main/en/version3.x/pipeline_usage/PP-StructureV3.html>
- PaddleOCR-VL-1.6 docs:
<https://paddlepaddle.github.io/PaddleOCR/main/en/version3.x/algorithm/PaddleOCR-VL/PaddleOCR-VL-1.6.html>
- Unicode Limbu chart: <https://www.unicode.org/charts/PDF/U1900.pdf>
- Unicode Kirat Rai proposal:
<https://www.unicode.org/L2/L2022/22043-kirat-rai.pdf>
- Kirat Rai orthography notes:
<https://r12a.github.io/scripts/krai/bap>
- Unicode Devanagari core-spec chapter:
<https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-12/>
- Magar Akkha proposal:
<https://www.unicode.org/L2/L2011/11144-magar-akkha.pdf>