# Claude Handoff: Nepal Multilingual OCR Goal: build and publish a research-grade PaddleOCR/PP-Structure OCR engine for at least ten languages spoken in Nepal, using Gorkhapatra/Naya Nepal pages as real validation evidence, PDF-native and legacy-font extraction for weak labels, and script-aware synthetic data for training and ablation. Updated: 2026-06-16 Repo: `Ampixa/ocr-tech` Visibility: private at handoff time Branch: `main` This handoff is for a follow-on Claude/Codex agent. The user is steering a research-grade OCR project, not a demo. The target is a PaddleOCR-centered document OCR engine whose claims are tied to real validation and reproducible training data, not synthetic-only benchmarks. ## Operating Rules - Use `uv run ...` for project commands. - Use `apply_patch` for manual edits. - Do not commit generated corpora, rendered images, model runs, or outputs unless explicitly requested; `data/`, `outputs/`, and `runs/` are ignored intentionally. - Do commit reproducibility configs, docs, tests, and source code changes. - GPG signing fails on this machine; use: ```bash git -c commit.gpgsign=false commit -m "" ``` - Do not commit remote passwords or credentials. Hosts are documented below, but credentials must stay in the conversation/session, not tracked files. - Do not claim SOTA or model quality unless backed by a passed real validation audit and benchmark. - Synthetic OCR data is never claim evidence. It is training/ablation material. - Validate schemas before writing converters. Do not infer fields from file names. ## Current Research Goal Build a PaddleOCR/PP-OCR + PP-Structure-style document engine for at least ten languages spoken in Nepal. Core scope: - OCR recognition - reading order - tables - figures/captions - Markdown output - JSON output - optional VLM fallback for hard figures/pages Current research angle: > Low-resource Nepal language OCR adaptation using PDF-native weak labels, > legacy font conversion, real Gorkhapatra validation, and script-aware > synthetic robustness. The real claim should be about measurable OCR/structure improvement on held-out Gorkhapatra/Naya Nepal pages, not about "understanding everything." ## Repo State Important recent commits: ```text 22454e2 Document cdjk language corpus migration 1f98430 Record migrated synthesis text sweep ``` The repo was checked with: ```bash jq . configs/cdjk-language-corpus-migration-v1.json >/dev/null git diff --check uv run pytest tests/test_synthesis_resources.py -q ``` All passed before this handoff was written. At handoff time, the only unrelated visible untracked path was: ```text .playwright-mcp/ ``` That is browser scratch output and should not be committed. ## Most Important Existing Docs Read these before acting: - `docs/ACTIVE_RESEARCH_CHECKLIST.md` - `docs/GORKHAPATRA_LANGUAGE_PAGE_ALIGNMENT.md` - `docs/CDJK_LANGUAGE_CORPUS_MIGRATION.md` - `docs/SYNTHESIS_RESOURCE_AUDIT.md` - `docs/FONT_ASSETS.md` - `docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md` - `docs/NEPAL_MULTILINGUAL_OCR_GOAL.md` ## What Has Been Built ### CLI Surface The repo already has many `ocrtech` commands. Most relevant now: ```text ocrtech audit-gorkhapatra-language-pages ocrtech prepare-gorkhapatra-language-page-pack ocrtech prepare-gorkhapatra-language-page-review ocrtech audit-gorkhapatra-language-page-review ocrtech extract-gorkhapatra-language-page-pdf-text ocrtech convert-limbu-legacy ocrtech audit-synthesis-resources ocrtech prepare-limbu-limdic-text ocrtech prepare-limbu-unicode-text ocrtech prepare-tamang-text ocrtech prepare-magar-text ocrtech prepare-toolkit-parallel-text ocrtech audit-synthesis-text-manifest ocrtech split-synthesis-text ocrtech render-synthesis-text-lines ocrtech render-synthesis-text-split ocrtech audit-rendered-synthesis-lines ocrtech audit-rendered-degradation-split ocrtech train-recognizer ``` ### Limbu PDF-Native Extraction Source rule for every Gorkhapatra/Naya Nepal language: `https://gorkhapatraonline.com/publications/naya-nepal` is only the discovery index for identifying which language/date was posted. Do not use the Naya Nepal blog/card image as OCR, eval, or training evidence. Fetch the full paper from the Gorkhapatra epaper date-range endpoint, for example `https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-16&pdfdaterange2=2026-06-23&slug=gorkhapatra`, then render the matching PDF page and extract/convert the text layer from that same PDF page. Record PDF URL/hash/page number, rendered image hash, text extraction provenance, and conversion map hash. Classify each language page as `unicode_native`, `legacy_font`, `devanagari_transcription`, `latin_notation`, or `no_text_layer`; manual transcription/native-reader work is only the fallback when the PDF has no usable text layer or the font cannot yet be decoded. The current successful Gorkhapatra Limbu path: ```bash uv run ocrtech extract-gorkhapatra-language-page-pdf-text \ data/limbu-gorkhapatra-language-page-pack-2026-06-16/manifests/gorkhapatra-language-page-pack.jsonl \ --candidate-language लिम्बू \ --out data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16 ``` Known result: ```text passed=True samples=2/2 spans=776 target_spans=304 fonts=10 warnings=0 sample_id=gorkhapatra-language-page-0001-01 pdf_page_number=6 target_font=NamdhinggoSILL target_spans=304 embedded_font=UFNWLJ+NamdhinggoSILL ``` Page conclusion: - page 6 is the Limbu/Sirijonga page - page 7 is the same-date non-Limbu page The PDF text layer exposes legacy ASCII-like Namdhinggo text, not Unicode Limbu. Example: ```text legacy: kfMG g' ;fK;SF[ yf]af]cf unicode: ᤐᤠ᤺ᤴ ᤏᤢ ᤛᤠᤵᤛᤰᤖᤧ ᤌᤥᤒᤥᤀᤠ ``` The converter is implemented in `src/ocrtech/limbu_legacy.py` using the vendored SIL map: ```text data/mappings/limbu-legacy/Limbu.map ``` ### Limbu Review Dashboard Generated dashboard path: ```text data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html ``` Correct local file URL: ```text file:///Users/cdjk/github/llm/ocr-tech/data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html ``` This is the correct browser UI for the Gorkhapatra PDF-native Limbu page. It shows the full rendered PDF image with bounded boxes; clicking a box selects the matching right-side row with converted text and the corresponding rendered span crop. Do not substitute the PDF-native verification-bundle HTML for this task; that was the wrong/broken UI. Local server used: ```bash uv run python -m http.server 8765 --bind 127.0.0.1 ``` Dashboard URL: ```text http://127.0.0.1:8765/data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html ``` Important bug already fixed in the generated dashboard: clicking a bounding box must not re-render or scroll the image/page back to the top. The fix removed automatic `scrollIntoView` from bounding-box selection. If this dashboard is regenerated by code later, preserve this behavior: ```js function selectSpan(id){ selected=id; document.querySelectorAll('.box.selected,.span.selected').forEach(el => el.classList.remove('selected')); document.querySelectorAll(`[data-span="${id}"],[data-row="${id}"]`).forEach(el => el.classList.add('selected')); } ``` ### Migrated Corpus Source host: ```text cdjk@100.110.148.43 machine: cdjks-MacBook-Pro.local external volume: /Volumes/TRANSCEND ``` Do not write generated data back to that host. Treat it as a source for corpus and dictionaries unless the user explicitly asks otherwise. Tracked migration policy: ```text configs/cdjk-language-corpus-migration-v1.json ``` Ignored local copy: ```text data/external-language-resources/cdjk-100.110.148.43/ ``` Copied roots: ```text dhimal-toolkit gurung-language-toolkit limbu-speech-toolkit ltk magar-toolkit rai-speech-toolkit ``` Migration result: ```text 2307 text/resource files transferred 1.6G local size resource audit: passed=True roots=6/6 files=2308 warnings=0 ``` ### Prepared Synthesis Text Generated synthesis text is ignored under `data/synthesis-text/`, but the documented state is: ```text 20 manifests 228212 total samples 45258 duplicate rows removed/reported by converters 202560 Devanagari rows 10337 Mixed_Devanagari_Latin rows 6521 Latin rows 8772 true Unicode Limbu rows 14 Mixed_Limbu_Devanagari rows 3 Mixed_Limbu_Latin rows 5 Other rows ``` Important generated manifests: ```text data/synthesis-text/cdjk-limbu-limdic-2026-06-16/limbu-limdic-text.jsonl data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered/limbu-only.jsonl data/synthesis-text/cdjk-tamang-2026-06-16/tamang-text.jsonl data/synthesis-text/cdjk-magar-2026-06-16/magar-text.jsonl data/synthesis-text/cdjk-gurung-2026-06-16/toolkit-parallel-text.jsonl data/synthesis-text/cdjk-dhimal-parallel-2026-06-16/toolkit-parallel-text.jsonl data/synthesis-text/cdjk-dhimal-mono-2026-06-16/toolkit-parallel-text.jsonl data/synthesis-text/cdjk-magar-parallel-extra-2026-06-16/toolkit-parallel-text.jsonl data/synthesis-text/cdjk-magar-mono-extra-2026-06-16/toolkit-parallel-text.jsonl data/synthesis-text/cdjk-rai-*-2026-06-16/toolkit-parallel-text.jsonl ``` All synthesis text audits passed with `claim_eligible=0`. Important interpretation: - True Unicode Limbu exists from Limdic and should render with Namdhinggo after filtering `script == Limbu`. - Most Gurung, Dhimal, Magar, Tamang, and Rai rows are Devanagari-transcribed language text, not native-script OCR. - Bantawa, Khaling, Koyu, and Thulung include substantial Latin phonetic data. These are useful for language resources but should not be called native-script OCR data. ### Synthetic Rendering Corrected Limbu-only smoke render: ```bash uv run ocrtech render-synthesis-text-lines \ data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered/limbu-only.jsonl \ --out data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only \ --font-path data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-Regular.ttf \ --limit 120 \ --degradation-profile clean \ --degradation-profile scan \ --degradation-profile low_light \ --degradation-profile uneven_light \ --degradation-profile camera_left \ --degradation-profile phone_photo ``` Audit: ```bash uv run ocrtech audit-rendered-synthesis-lines \ data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only/manifests/rendered-line-manifest.jsonl \ --label-file data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only/labels/train.txt \ --out outputs/cdjk-limbu-rendered-smoke-audit-2026-06-16-limbu-only ``` Result: ```text passed=True samples=120 claim_eligible=0 issues=0 warnings=0 degradation_counts: camera_left: 20 clean: 20 low_light: 20 phone_photo: 20 scan: 20 uneven_light: 20 ``` Important lesson: an earlier mixed-script render produced tofu boxes because Devanagari/Latin rows were rendered with a Limbu font. Always filter by script before choosing a font. ## Current Strategic Decision The user asked what next. The right order is: 1. Build Gorkhapatra validation for all target languages. 2. Create language/script/font matrix from actual Gorkhapatra PDF evidence. 3. Generate split-safe synthetic data per script/font bucket. 4. Research and execute PaddleOCR recognizer fine-tuning. 5. Keep PP-Structure/layout mostly stock until recognizer evidence says detection/layout fine-tuning is required. Do not scale synthetic data first without real validation. It will create a large training set but no credible paper claim. ## Immediate Next Task Build a reusable all-language Gorkhapatra validation generator. It should turn this manual Limbu success into a repeatable command/workflow: ```text Naya Nepal publication card -> date/language/article metadata -> matching epaper PDF -> candidate language-page pages -> rendered page image -> PDF-native spans/fonts/bboxes/codepoints -> optional legacy conversion by script/font -> review dashboard -> reviewed JSON/CSV annotations ``` Suggested CLI shape: ```text ocrtech prepare-gorkhapatra-language-validation ``` Likely arguments: ```text --language-label --publication-pages --epaper-url --out --target-font --legacy-converter --render-pages --download-pdfs ``` Output should include: ```text validation.json validation.md samples/*.pdf-text.json samples/*.pdf-text.txt spans.csv fonts/ page-images/ span-renders/ dashboard.html review.csv review.jsonl ``` Dashboard requirements: - Full page image with clickable bbox overlay. - Right panel with selected span, PDF-rendered crop, raw extracted text, converted Unicode text when available, font, bbox, codepoints. - Review status controls: - `accept` - `needs_fix` - `bad_segment` - `drop` - `wrong_page` - `wrong_language` - Must preserve image/page scroll when clicking bbox. - Must not ask the user to transcribe text if PDF-native extraction/conversion already produces usable Unicode text. - Must save/export annotations as JSONL/CSV. ## Language Validation Targets At least ten target languages are desired. Current likely sources: - Limbu/Sirijonga - Tamang - Magar - Gurung - Dhimal - Newar/Newa - Bantawa/Rai - Chamling/Rai - Kulung/Rai - Athpariya/Rai - Khaling/Rai - Yakkha/Rai - Sunwar/Rai - Lohorung/Rai - Dumi/Rai - Thulung/Rai - Koyu/Rai But Gorkhapatra validation must decide actual script form: - Unicode native script - legacy font encoding - Devanagari transcription - Latin notation - unsupported/no text layer Do not assume native Unicode just because the language has a script. ## Training Research Direction Use primary sources before changing training code. Current understanding: - PaddleOCR/PP-OCR remains the primary engine family. - Start by fine-tuning recognizer, not layout/detection. - Use PP-Structure/PP-StructureV3-style parsing for document layout and structured output. - Consider PaddleOCR-VL only as fallback/teacher/comparator, not the default small local engine. Recognizer-first rationale: - The repo already has line/synthesis text and rendered line labels. - Recognition turns cropped text images into text. - Detection/layout claims require bbox/layout ground truth, which is not yet mature across languages. - Fine-tuning detection before measuring detection failures is premature. Training prerequisites before a serious run: - Split synthesis text with `split-synthesis-text`. - Exclude real validation/reference text hashes from synthetic training. - Render train/eval with font readiness and degradation audits. - Keep English/Nepali non-regression sets. - Package PaddleOCR recognizer training recipe. - Evaluate on Gorkhapatra real pages and held-out synthetic rows. ## Remote Machines Known hosts from the session: ```text k2@100.97.238.103 cdjk@100.110.148.43 cdjk@100.117.21.47 ``` Do not commit passwords. The user provided credentials in chat previously, but tracked docs must not contain them. Important storage findings: - `cdjk@100.110.148.43` has `/Volumes/TRANSCEND`, about `3.6Ti` total and `2.0Ti` available at inventory time. Use as source for corpus/dictionary resources unless instructed otherwise. - `cdjk@100.117.21.47` root was full at inspection time: ```text / 915G 865G 3.7G 100% /mnt/tts_cache_nvme 234G 150G 72G 68% ``` Do not dump large generated data to a nearly-full root filesystem. ## Source Files To Inspect Before Editing For Gorkhapatra flow: ```text src/ocrtech/source_audit.py src/ocrtech/cli.py src/ocrtech/limbu_legacy.py tests/test_source_audit.py tests/test_limbu_legacy.py ``` For synthesis: ```text src/ocrtech/synthesis_resources.py tests/test_synthesis_resources.py configs/cdjk-language-corpus-migration-v1.json ``` For fonts/language registry: ```text src/ocrtech/font_assets.py src/ocrtech/language_registry.py configs/namdhinggo-font-assets-v1.json configs/nepal-font-assets-v1.json languages/nepal-v1.json ``` For training: ```text src/ocrtech/training.py src/ocrtech/models.py docs/MODELS.md docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md ``` ## Known Pitfalls - The user is frustrated by fake OCR candidates and manual busywork. Do not ask them to validate bad OCR text when PDF-native text extraction can provide the answer. - For Limbu, PDF text was extractable but legacy encoded. The correct path was PDF-native text -> Namdhinggo legacy conversion -> Unicode display. - Dashboard bbox clicks must not reset scroll. - Do not treat Devanagari-transcribed language data as native script OCR. - Do not render mixed-script rows with a single script-specific font. - Do not collapse mixed schemas into one converter command by dropping validation. Split sources by schema instead. - Do not publish synthetic-only claims. ## Recommended Next Commit Implement the reusable validation generator and tests: 1. Refactor current generated dashboard logic into source code. 2. Add a CLI command that accepts a manifest/PDF-text extraction dir and emits dashboard + review CSV/JSONL. 3. Preserve current Limbu dashboard features: - full page image - bbox overlay - PDF-rendered span crop - converted Unicode text - raw legacy text - font metadata - stable scroll behavior 4. Add fixture tests for dashboard generation and annotation schema. 5. Document command in `docs/GORKHAPATRA_LANGUAGE_PAGE_ALIGNMENT.md`. 6. Run focused tests: ```bash uv run pytest tests/test_source_audit.py tests/test_limbu_legacy.py -q git diff --check ``` Then commit with GPG signing disabled and push.