File size: 14,177 Bytes
f9bc0c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 | # Multilingual PaddleOCR Research Direction
This note records the pivot from a Nepali/English recognizer experiment to a
PaddleOCR-centered low-resource document OCR program for Nepal's languages. The
research claim should be edited before more GPU time is spent: we are no longer
trying to beat the world with a tiny Nepali line dataset. The credible angle is
script-aware, low-resource adaptation of PaddleOCR for multilingual Nepali
documents, with document-structure validation strong enough to reject weak
claims.
## Current Decision
PaddleOCR should be the primary engine family.
The reason is practical, not ideological:
- PP-OCRv6 is now the strongest small/local PaddleOCR text OCR direction, with
official tiny/small/medium tiers and a unified multilingual recognizer for
Chinese, English, Japanese, and Latin-script languages.
- PP-StructureV3 is the right local document-parsing scaffold for layout,
tables, reading order, formulas, seals, and document reconstruction.
- PaddleOCR-VL-1.6 is the right fallback/teacher for hard pages, charts,
figures, seals, and pages where a small local pipeline has low confidence.
- PaddleOCR already has the training/deployment stack we need: dictionaries,
recognition configs, layout modules, table modules, model export, and CPU/GPU
deployment paths.
The caveat is important: PP-OCRv6's headline multilingual coverage does not
solve Limbu, Kirat Rai, Tamang Tamyig, or other Nepal-specific scripts by
default. We should use PaddleOCR as the framework and base model family, then
train or adapt script-specific recognition heads/dictionaries where needed.
## Revised Research Thesis
Candidate thesis:
> We build a PaddleOCR-based multilingual document OCR engine for Nepali,
> English, and low-resource languages of Nepal by combining script-aware corpus
> construction, Unicode-safe synthetic document generation, PaddleOCR
> recognizer fine-tuning, PP-Structure-style document parsing, and selective
> VLM fallback. We evaluate against Tesseract, stock PaddleOCR/PP-OCR, Surya,
> GLM-OCR, and PaddleOCR-VL on OCR accuracy, reading order, table structure,
> figure/caption handling, and Markdown/JSON reconstruction.
This is stronger than "Nepali OCR beats Tesseract" because it contributes:
- a multilingual/script inventory for Nepal document OCR
- a reproducible synthetic document generator grounded in real text resources
- a PaddleOCR adaptation recipe for low-resource scripts/languages
- held-out public plus real-document validation rather than synthetic-only
claims
- a clear local engine plus optional VLM fallback design
## Language And Script Inventory
OCR support should be tracked by `(language, script, Unicode, font, corpus,
evaluation)` rather than by language name alone.
| Target | Likely script(s) | Unicode status | Current local resources found | OCR implication |
| --- | --- | --- | --- | --- |
| Nepali | Devanagari | U+0900..U+097F plus extensions as needed | Existing OCR manifests, G2P/text frontend, Loksewa/Gorkhapatra-like documents | Keep as core training/eval language |
| English | Latin | Basic/extended Latin | NVIDIA synthetic line crops, document text | Preserve English, do not regress |
| Tamang | Mostly Devanagari in local corpora; Tamyig/Tibetan-related script also exists | Devanagari is supported; Tamyig needs separate font/encoding audit | `~/github/llm/ltk/tamang`: NepTam train/test, NepTam80K synthetic, dictionary, TTS lexicon | Start with Devanagari Tamang; audit Tamyig separately |
| Limbu/Yakthung | Limbu/Sirijonga | Limbu block U+1900..U+194F | `~/github/llm/ltk/limbu`: Limbu dictionary, Pangloss IGT, lexicon, G2P | Needs Limbu fonts, Paddle dictionary, synthetic rendering, held-out real pages |
| Rai/Bantawa/Kirat Rai | Kirat Rai and Devanagari/Latin transcriptions depending source | Kirat Rai block U+16D40..U+16D7F in Unicode 16.0 | No local corpus confirmed yet; web sources point to Sikkim Herald/Rai textbooks/proposal examples | Needs corpus acquisition and fonts before training |
| Magar | Primarily Devanagari; Magar Akkha is a community-script issue | Devanagari supported; Magar Akkha status is conflicting in web sources and must be verified against the current Unicode database before use | `~/github/llm/ltk/magar`: Western Magar dictionary and resource report | Start with Devanagari Magar; treat Akkha as research/audit item |
| Nepal Bhasa/Newar | Newar/Prachalit, Bhujimol, Ranjana, and Devanagari | Prachalit uses Newa U+11400..U+1147F; Bhujimol has no separate script encoding and uses an explicit Newa-transcription style profile; Ranjana remains unencoded; Devanagari is separate | Four routes are structurally audited; Prachalit and Devanagari have diagnostic synthetic runtime smokes, Bhujimol and Ranjana have exact reviewer references and acquired review candidates, and one permission-covered official Newa edition is cataloged for remote script screening | Keep four visual profiles separate; references and acquisition candidates are not recognizer-quality evidence, and shared transcription codepoints do not imply visual-model coverage |
Do not call these "dialects" in the paper unless the linguistic unit really is
a dialect. OCR sees scripts and glyph distributions first. Tamang varieties,
Rai languages, Magar varieties, and Limbu are not one generic dialect bucket.
## Local Resource Inventory
Verified local files on this machine:
### Tamang
Directory: `~/github/llm/ltk/tamang`
- `data/neptam_train.csv`: 15,003 lines including header.
- `data/neptam_test.csv`: 5,003 lines including header.
- `data/neptam80k_synthetic.csv`: 80,100 lines including header.
- `data/western_tamang_dictionary.tsv`: 3,605 lines including header.
- `data/tts_lexicon.tsv`: 107,225 lines.
Observed fields:
- `neptam_train.csv`: `sentence_id,nepali_sentences,translation_tamang,sentence_type,Tense,polarity`
- `western_tamang_dictionary.tsv`: `headword,pos,definition`
This is enough to synthesize Devanagari Tamang/Nepali line crops and document
pages immediately, subject to license review.
### Limbu
Directory: `~/github/llm/ltk/limbu`
- `data/limdic_dictionary.tsv`: 8,798 lines including header.
- `data/pangloss_dictionary.tsv`: 2,541 lines including header.
- `data/pangloss_igt_corpus.tsv`: 10,951 lines including header.
- `data/pangloss_nepali_loanwords.tsv`: 353 lines including header.
- `data/tts_lexicon.tsv`: 2,541 lines including header.
- `g2p/limbu_g2p.py`: Unicode Limbu to phoneme tooling.
Observed fields:
- `limdic_dictionary.tsv`: `id,headword_limbu,desc_devanagari,group,definition,ipa`
- `pangloss_dictionary.tsv`: `headword,pos,definitions,nepali,etymology,root,example_count`
This is enough for first-pass Limbu synthetic line generation, but it needs
font and rendering validation before recognizer training.
### Magar
Directory: `~/github/llm/ltk/magar`
- `data/western_magar_dictionary.tsv`: 583 lines including header.
- `data/magar_resource_report.md`: source survey.
Observed fields:
- `western_magar_dictionary.tsv`: `headword,ipa,pos,variant,definition`
This is too small for a standalone Magar OCR claim, but useful as seed text for
synthetic rendering and dictionary coverage tests.
### Other Nepal-Focused Sources
Potentially useful local directories:
- `~/github/llm/loksewa`: many Nepali PDFs, PNGs, and Markdown documents with
government-exam style layouts.
- `~/github/llm/g2p` and `~/github/llm/nepali_text_frontend`: Nepali text
normalization, lexicon, and akshara resources.
- `~/github/llm/new-place`: Nepal-focused scraped text archives.
The requested remote host `cdjk@100.110.148.43` is reachable, but this machine
does not have `sshpass`. An `expect` attempt reached the password prompt but
did not complete cleanly in this run. Remote corpus inventory remains pending.
## Data Synthesis Strategy
Synthesize documents, not just isolated words.
### 1. Text Layer
Use real text wherever possible:
- Nepali: local G2P/frontend corpora, Loksewa/government-style pages, public
domain/news/government sources after license audit.
- Tamang: NepTam train/test and dictionary material.
- Limbu: Unicode Limbu dictionary entries, Pangloss-derived text, loanwords,
and IGT corpus rows.
- Magar: Western Magar dictionary entries and later Webonary/app-derived data
only if legally extractable.
- Rai/Bantawa/Kirat Rai: Gorkhapatra/Naya Nepal, Sikkim Herald/Rai materials,
textbooks, and Unicode proposal examples only after acquisition and license
review.
### 2. Script And Font Layer
For every target script, create a renderability audit:
- Unicode block coverage.
- Fonts available on macOS/Linux/AWS.
- Shaping correctness with HarfBuzz/Pango.
- Unsupported character list.
- Confusable/normalization policy.
- PaddleOCR dictionary file.
- Grapheme/akshara segmentation tests.
No sample should enter training unless its text can be rendered and normalized
deterministically.
### 3. OCR Crop Generation
Generate:
- word crops
- line crops
- table-cell crops
- caption crops
- noisy crops extracted from rendered pages
For PaddleOCR training, each generated crop needs:
- image path
- exact UTF-8 label
- script/language metadata
- source corpus id
- source text hash
- render template id
- font id
- degradation id
### 4. Full Document Generation
Render complete pages with ground truth:
- single-column and two-column articles
- school notices
- government forms
- invoices/receipts
- Naya Nepal/Gorkhapatra-style multilingual pages
- tables with merged cells
- figure/chart blocks with captions
- headers/footers/page numbers
For every page emit:
- `document.json` reference with blocks, lines, reading order, tables, figures,
captions, and bboxes
- `document.md` reference
- table CSV/HTML references
- crop manifests for recognizer training
### 5. Degradation Layer
Apply realistic scan/photo transforms:
- blur
- skew
- rotation
- low contrast
- JPEG compression
- bleed-through/noise
- perspective warp
- uneven lighting
- cropped margins
- phone-photo shadows
The degradation must be recorded in metadata so evaluation can report which
conditions fail.
## PaddleOCR Training Plan
Recommended branches:
1. **PP-OCRv6 baseline audit**
- Run stock PP-OCRv6 on our current Nepali/English eval packs.
- Add Tamang/Limbu smoke pages after fonts render correctly.
- Record failures by script and degradation.
2. **Devanagari multilingual recognizer**
- Fine-tune PaddleOCR recognizer for Nepali + English + Devanagari Tamang +
Devanagari Magar/Rai where text exists.
- Dictionary: Devanagari, Latin, punctuation, Devanagari digits, Arabic
digits, symbols used in forms/tables.
- Gate: must beat stock PP-OCRv6/PP-OCRv5 and Tesseract on the same
Devanagari multilingual slices without English regression.
3. **Limbu recognizer**
- Build a separate Limbu/Sirijonga dictionary and synthetic corpus.
- Start with line/cell OCR, then document pages.
- Gate: beat stock PaddleOCR fallback behavior and any available Tesseract
baseline if traineddata exists; otherwise compare to PaddleOCR-VL and
report as new low-resource benchmark rather than broad SOTA.
4. **Kirat Rai recognizer**
- Do not train until corpus/font audit is complete.
- Unicode 16.0 support means runtime/font stacks may lag; verify first.
5. **PP-Structure integration**
- Keep the document pipeline engine-side: layout, reading order, table
reconstruction, figure/caption handling, Markdown/JSON export.
- Replace recognizer component per script/domain.
- Use PaddleOCR-VL only as selective fallback or teacher, not as the default
"small local" engine.
## Validation Plan
The revised claim needs these evaluation groups:
1. **Recognizer OCR**
- CER/WER per language.
- CER/WER per script.
- Exact-line accuracy.
- Unicode-normalized and strict metrics.
- English non-regression.
2. **Document Understanding**
- Reading order.
- Markdown reconstruction.
- Table cell text and structure.
- HTML/TEDS-style table similarity.
- Figure/caption detection.
- Required output artifact checks.
3. **Baselines**
- Tesseract where language/script data exists.
- Stock PaddleOCR PP-OCRv5/PP-OCRv6.
- Surya.
- GLM-OCR.
- PaddleOCR-VL-1.6.
4. **Splits**
- Synthetic train.
- Synthetic held-out by source text/template/font/degradation.
- Real held-out documents.
- Public benchmarks separate from all training.
No SOTA claim should be accepted from synthetic-only evaluation. Synthetic data
is for scale and controlled ablations. Real and public held-out data are for
claims.
## Immediate Engineering Tasks
1. Add a `language_registry` file describing language, script, Unicode ranges,
required fonts, normalization policy, and Paddle dictionary path.
2. Add corpus importers for the verified local Tamang, Limbu, and Magar files.
3. Add a renderability audit command that fails on unsupported characters,
missing glyphs, bad shaping, or normalization drift.
4. Add a synthetic line/page generator that emits PaddleOCR-compatible labels
plus document references.
5. Run a stock PP-OCRv6 baseline audit before the next fine-tune.
6. Only then start CUDA fine-tuning.
## Source Pointers
- PaddleOCR docs index, including PP-OCRv6, PP-StructureV3, and PaddleOCR-VL:
<https://paddlepaddle.github.io/PaddleOCR/main/en/index.html>
- PaddleOCR repository: <https://github.com/PaddlePaddle/PaddleOCR>
- PaddleOCR PP-StructureV3 pipeline docs:
<https://paddlepaddle.github.io/PaddleOCR/main/en/version3.x/pipeline_usage/PP-StructureV3.html>
- PaddleOCR-VL-1.6 docs:
<https://paddlepaddle.github.io/PaddleOCR/main/en/version3.x/algorithm/PaddleOCR-VL/PaddleOCR-VL-1.6.html>
- Unicode Limbu chart: <https://www.unicode.org/charts/PDF/U1900.pdf>
- Unicode Kirat Rai proposal:
<https://www.unicode.org/L2/L2022/22043-kirat-rai.pdf>
- Kirat Rai orthography notes:
<https://r12a.github.io/scripts/krai/bap>
- Unicode Devanagari core-spec chapter:
<https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-12/>
- Magar Akkha proposal:
<https://www.unicode.org/L2/L2011/11144-magar-akkha.pdf>
|