|
Download ocr-tech/docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 14.2 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md
-
curl -L -o MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md
14.2 kB
| # Multilingual PaddleOCR Research Direction | |
| This note records the pivot from a Nepali/English recognizer experiment to a | |
| PaddleOCR-centered low-resource document OCR program for Nepal's languages. The | |
| research claim should be edited before more GPU time is spent: we are no longer | |
| trying to beat the world with a tiny Nepali line dataset. The credible angle is | |
| script-aware, low-resource adaptation of PaddleOCR for multilingual Nepali | |
| documents, with document-structure validation strong enough to reject weak | |
| claims. | |
| ## Current Decision | |
| PaddleOCR should be the primary engine family. | |
| The reason is practical, not ideological: | |
| - PP-OCRv6 is now the strongest small/local PaddleOCR text OCR direction, with | |
| official tiny/small/medium tiers and a unified multilingual recognizer for | |
| Chinese, English, Japanese, and Latin-script languages. | |
| - PP-StructureV3 is the right local document-parsing scaffold for layout, | |
| tables, reading order, formulas, seals, and document reconstruction. | |
| - PaddleOCR-VL-1.6 is the right fallback/teacher for hard pages, charts, | |
| figures, seals, and pages where a small local pipeline has low confidence. | |
| - PaddleOCR already has the training/deployment stack we need: dictionaries, | |
| recognition configs, layout modules, table modules, model export, and CPU/GPU | |
| deployment paths. | |
| The caveat is important: PP-OCRv6's headline multilingual coverage does not | |
| solve Limbu, Kirat Rai, Tamang Tamyig, or other Nepal-specific scripts by | |
| default. We should use PaddleOCR as the framework and base model family, then | |
| train or adapt script-specific recognition heads/dictionaries where needed. | |
| ## Revised Research Thesis | |
| Candidate thesis: | |
| > We build a PaddleOCR-based multilingual document OCR engine for Nepali, | |
| > English, and low-resource languages of Nepal by combining script-aware corpus | |
| > construction, Unicode-safe synthetic document generation, PaddleOCR | |
| > recognizer fine-tuning, PP-Structure-style document parsing, and selective | |
| > VLM fallback. We evaluate against Tesseract, stock PaddleOCR/PP-OCR, Surya, | |
| > GLM-OCR, and PaddleOCR-VL on OCR accuracy, reading order, table structure, | |
| > figure/caption handling, and Markdown/JSON reconstruction. | |
| This is stronger than "Nepali OCR beats Tesseract" because it contributes: | |
| - a multilingual/script inventory for Nepal document OCR | |
| - a reproducible synthetic document generator grounded in real text resources | |
| - a PaddleOCR adaptation recipe for low-resource scripts/languages | |
| - held-out public plus real-document validation rather than synthetic-only | |
| claims | |
| - a clear local engine plus optional VLM fallback design | |
| ## Language And Script Inventory | |
| OCR support should be tracked by `(language, script, Unicode, font, corpus, | |
| evaluation)` rather than by language name alone. | |
| | Target | Likely script(s) | Unicode status | Current local resources found | OCR implication | | |
| | --- | --- | --- | --- | --- | | |
| | Nepali | Devanagari | U+0900..U+097F plus extensions as needed | Existing OCR manifests, G2P/text frontend, Loksewa/Gorkhapatra-like documents | Keep as core training/eval language | | |
| | English | Latin | Basic/extended Latin | NVIDIA synthetic line crops, document text | Preserve English, do not regress | | |
| | Tamang | Mostly Devanagari in local corpora; Tamyig/Tibetan-related script also exists | Devanagari is supported; Tamyig needs separate font/encoding audit | `~/github/llm/ltk/tamang`: NepTam train/test, NepTam80K synthetic, dictionary, TTS lexicon | Start with Devanagari Tamang; audit Tamyig separately | | |
| | Limbu/Yakthung | Limbu/Sirijonga | Limbu block U+1900..U+194F | `~/github/llm/ltk/limbu`: Limbu dictionary, Pangloss IGT, lexicon, G2P | Needs Limbu fonts, Paddle dictionary, synthetic rendering, held-out real pages | | |
| | Rai/Bantawa/Kirat Rai | Kirat Rai and Devanagari/Latin transcriptions depending source | Kirat Rai block U+16D40..U+16D7F in Unicode 16.0 | No local corpus confirmed yet; web sources point to Sikkim Herald/Rai textbooks/proposal examples | Needs corpus acquisition and fonts before training | | |
| | Magar | Primarily Devanagari; Magar Akkha is a community-script issue | Devanagari supported; Magar Akkha status is conflicting in web sources and must be verified against the current Unicode database before use | `~/github/llm/ltk/magar`: Western Magar dictionary and resource report | Start with Devanagari Magar; treat Akkha as research/audit item | | |
| | Nepal Bhasa/Newar | Newar/Prachalit, Bhujimol, Ranjana, and Devanagari | Prachalit uses Newa U+11400..U+1147F; Bhujimol has no separate script encoding and uses an explicit Newa-transcription style profile; Ranjana remains unencoded; Devanagari is separate | Four routes are structurally audited; Prachalit and Devanagari have diagnostic synthetic runtime smokes, Bhujimol and Ranjana have exact reviewer references and acquired review candidates, and one permission-covered official Newa edition is cataloged for remote script screening | Keep four visual profiles separate; references and acquisition candidates are not recognizer-quality evidence, and shared transcription codepoints do not imply visual-model coverage | | |
| Do not call these "dialects" in the paper unless the linguistic unit really is | |
| a dialect. OCR sees scripts and glyph distributions first. Tamang varieties, | |
| Rai languages, Magar varieties, and Limbu are not one generic dialect bucket. | |
| ## Local Resource Inventory | |
| Verified local files on this machine: | |
| ### Tamang | |
| Directory: `~/github/llm/ltk/tamang` | |
| - `data/neptam_train.csv`: 15,003 lines including header. | |
| - `data/neptam_test.csv`: 5,003 lines including header. | |
| - `data/neptam80k_synthetic.csv`: 80,100 lines including header. | |
| - `data/western_tamang_dictionary.tsv`: 3,605 lines including header. | |
| - `data/tts_lexicon.tsv`: 107,225 lines. | |
| Observed fields: | |
| - `neptam_train.csv`: `sentence_id,nepali_sentences,translation_tamang,sentence_type,Tense,polarity` | |
| - `western_tamang_dictionary.tsv`: `headword,pos,definition` | |
| This is enough to synthesize Devanagari Tamang/Nepali line crops and document | |
| pages immediately, subject to license review. | |
| ### Limbu | |
| Directory: `~/github/llm/ltk/limbu` | |
| - `data/limdic_dictionary.tsv`: 8,798 lines including header. | |
| - `data/pangloss_dictionary.tsv`: 2,541 lines including header. | |
| - `data/pangloss_igt_corpus.tsv`: 10,951 lines including header. | |
| - `data/pangloss_nepali_loanwords.tsv`: 353 lines including header. | |
| - `data/tts_lexicon.tsv`: 2,541 lines including header. | |
| - `g2p/limbu_g2p.py`: Unicode Limbu to phoneme tooling. | |
| Observed fields: | |
| - `limdic_dictionary.tsv`: `id,headword_limbu,desc_devanagari,group,definition,ipa` | |
| - `pangloss_dictionary.tsv`: `headword,pos,definitions,nepali,etymology,root,example_count` | |
| This is enough for first-pass Limbu synthetic line generation, but it needs | |
| font and rendering validation before recognizer training. | |
| ### Magar | |
| Directory: `~/github/llm/ltk/magar` | |
| - `data/western_magar_dictionary.tsv`: 583 lines including header. | |
| - `data/magar_resource_report.md`: source survey. | |
| Observed fields: | |
| - `western_magar_dictionary.tsv`: `headword,ipa,pos,variant,definition` | |
| This is too small for a standalone Magar OCR claim, but useful as seed text for | |
| synthetic rendering and dictionary coverage tests. | |
| ### Other Nepal-Focused Sources | |
| Potentially useful local directories: | |
| - `~/github/llm/loksewa`: many Nepali PDFs, PNGs, and Markdown documents with | |
| government-exam style layouts. | |
| - `~/github/llm/g2p` and `~/github/llm/nepali_text_frontend`: Nepali text | |
| normalization, lexicon, and akshara resources. | |
| - `~/github/llm/new-place`: Nepal-focused scraped text archives. | |
| The requested remote host `cdjk@100.110.148.43` is reachable, but this machine | |
| does not have `sshpass`. An `expect` attempt reached the password prompt but | |
| did not complete cleanly in this run. Remote corpus inventory remains pending. | |
| ## Data Synthesis Strategy | |
| Synthesize documents, not just isolated words. | |
| ### 1. Text Layer | |
| Use real text wherever possible: | |
| - Nepali: local G2P/frontend corpora, Loksewa/government-style pages, public | |
| domain/news/government sources after license audit. | |
| - Tamang: NepTam train/test and dictionary material. | |
| - Limbu: Unicode Limbu dictionary entries, Pangloss-derived text, loanwords, | |
| and IGT corpus rows. | |
| - Magar: Western Magar dictionary entries and later Webonary/app-derived data | |
| only if legally extractable. | |
| - Rai/Bantawa/Kirat Rai: Gorkhapatra/Naya Nepal, Sikkim Herald/Rai materials, | |
| textbooks, and Unicode proposal examples only after acquisition and license | |
| review. | |
| ### 2. Script And Font Layer | |
| For every target script, create a renderability audit: | |
| - Unicode block coverage. | |
| - Fonts available on macOS/Linux/AWS. | |
| - Shaping correctness with HarfBuzz/Pango. | |
| - Unsupported character list. | |
| - Confusable/normalization policy. | |
| - PaddleOCR dictionary file. | |
| - Grapheme/akshara segmentation tests. | |
| No sample should enter training unless its text can be rendered and normalized | |
| deterministically. | |
| ### 3. OCR Crop Generation | |
| Generate: | |
| - word crops | |
| - line crops | |
| - table-cell crops | |
| - caption crops | |
| - noisy crops extracted from rendered pages | |
| For PaddleOCR training, each generated crop needs: | |
| - image path | |
| - exact UTF-8 label | |
| - script/language metadata | |
| - source corpus id | |
| - source text hash | |
| - render template id | |
| - font id | |
| - degradation id | |
| ### 4. Full Document Generation | |
| Render complete pages with ground truth: | |
| - single-column and two-column articles | |
| - school notices | |
| - government forms | |
| - invoices/receipts | |
| - Naya Nepal/Gorkhapatra-style multilingual pages | |
| - tables with merged cells | |
| - figure/chart blocks with captions | |
| - headers/footers/page numbers | |
| For every page emit: | |
| - `document.json` reference with blocks, lines, reading order, tables, figures, | |
| captions, and bboxes | |
| - `document.md` reference | |
| - table CSV/HTML references | |
| - crop manifests for recognizer training | |
| ### 5. Degradation Layer | |
| Apply realistic scan/photo transforms: | |
| - blur | |
| - skew | |
| - rotation | |
| - low contrast | |
| - JPEG compression | |
| - bleed-through/noise | |
| - perspective warp | |
| - uneven lighting | |
| - cropped margins | |
| - phone-photo shadows | |
| The degradation must be recorded in metadata so evaluation can report which | |
| conditions fail. | |
| ## PaddleOCR Training Plan | |
| Recommended branches: | |
| 1. **PP-OCRv6 baseline audit** | |
| - Run stock PP-OCRv6 on our current Nepali/English eval packs. | |
| - Add Tamang/Limbu smoke pages after fonts render correctly. | |
| - Record failures by script and degradation. | |
| 2. **Devanagari multilingual recognizer** | |
| - Fine-tune PaddleOCR recognizer for Nepali + English + Devanagari Tamang + | |
| Devanagari Magar/Rai where text exists. | |
| - Dictionary: Devanagari, Latin, punctuation, Devanagari digits, Arabic | |
| digits, symbols used in forms/tables. | |
| - Gate: must beat stock PP-OCRv6/PP-OCRv5 and Tesseract on the same | |
| Devanagari multilingual slices without English regression. | |
| 3. **Limbu recognizer** | |
| - Build a separate Limbu/Sirijonga dictionary and synthetic corpus. | |
| - Start with line/cell OCR, then document pages. | |
| - Gate: beat stock PaddleOCR fallback behavior and any available Tesseract | |
| baseline if traineddata exists; otherwise compare to PaddleOCR-VL and | |
| report as new low-resource benchmark rather than broad SOTA. | |
| 4. **Kirat Rai recognizer** | |
| - Do not train until corpus/font audit is complete. | |
| - Unicode 16.0 support means runtime/font stacks may lag; verify first. | |
| 5. **PP-Structure integration** | |
| - Keep the document pipeline engine-side: layout, reading order, table | |
| reconstruction, figure/caption handling, Markdown/JSON export. | |
| - Replace recognizer component per script/domain. | |
| - Use PaddleOCR-VL only as selective fallback or teacher, not as the default | |
| "small local" engine. | |
| ## Validation Plan | |
| The revised claim needs these evaluation groups: | |
| 1. **Recognizer OCR** | |
| - CER/WER per language. | |
| - CER/WER per script. | |
| - Exact-line accuracy. | |
| - Unicode-normalized and strict metrics. | |
| - English non-regression. | |
| 2. **Document Understanding** | |
| - Reading order. | |
| - Markdown reconstruction. | |
| - Table cell text and structure. | |
| - HTML/TEDS-style table similarity. | |
| - Figure/caption detection. | |
| - Required output artifact checks. | |
| 3. **Baselines** | |
| - Tesseract where language/script data exists. | |
| - Stock PaddleOCR PP-OCRv5/PP-OCRv6. | |
| - Surya. | |
| - GLM-OCR. | |
| - PaddleOCR-VL-1.6. | |
| 4. **Splits** | |
| - Synthetic train. | |
| - Synthetic held-out by source text/template/font/degradation. | |
| - Real held-out documents. | |
| - Public benchmarks separate from all training. | |
| No SOTA claim should be accepted from synthetic-only evaluation. Synthetic data | |
| is for scale and controlled ablations. Real and public held-out data are for | |
| claims. | |
| ## Immediate Engineering Tasks | |
| 1. Add a `language_registry` file describing language, script, Unicode ranges, | |
| required fonts, normalization policy, and Paddle dictionary path. | |
| 2. Add corpus importers for the verified local Tamang, Limbu, and Magar files. | |
| 3. Add a renderability audit command that fails on unsupported characters, | |
| missing glyphs, bad shaping, or normalization drift. | |
| 4. Add a synthetic line/page generator that emits PaddleOCR-compatible labels | |
| plus document references. | |
| 5. Run a stock PP-OCRv6 baseline audit before the next fine-tune. | |
| 6. Only then start CUDA fine-tuning. | |
| ## Source Pointers | |
| - PaddleOCR docs index, including PP-OCRv6, PP-StructureV3, and PaddleOCR-VL: | |
| <https://paddlepaddle.github.io/PaddleOCR/main/en/index.html> | |
| - PaddleOCR repository: <https://github.com/PaddlePaddle/PaddleOCR> | |
| - PaddleOCR PP-StructureV3 pipeline docs: | |
| <https://paddlepaddle.github.io/PaddleOCR/main/en/version3.x/pipeline_usage/PP-StructureV3.html> | |
| - PaddleOCR-VL-1.6 docs: | |
| <https://paddlepaddle.github.io/PaddleOCR/main/en/version3.x/algorithm/PaddleOCR-VL/PaddleOCR-VL-1.6.html> | |
| - Unicode Limbu chart: <https://www.unicode.org/charts/PDF/U1900.pdf> | |
| - Unicode Kirat Rai proposal: | |
| <https://www.unicode.org/L2/L2022/22043-kirat-rai.pdf> | |
| - Kirat Rai orthography notes: | |
| <https://r12a.github.io/scripts/krai/bap> | |
| - Unicode Devanagari core-spec chapter: | |
| <https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-12/> | |
| - Magar Akkha proposal: | |
| <https://www.unicode.org/L2/L2011/11144-magar-akkha.pdf> | |