ocr-projects-archive / ocr-tech /docs /CLAUDE_HANDOFF.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
17.8 kB
# Claude Handoff: Nepal Multilingual OCR
Goal: build and publish a research-grade PaddleOCR/PP-Structure OCR engine for
at least ten languages spoken in Nepal, using Gorkhapatra/Naya Nepal pages as
real validation evidence, PDF-native and legacy-font extraction for weak labels,
and script-aware synthetic data for training and ablation.
Updated: 2026-06-16
Repo: `Ampixa/ocr-tech`
Visibility: private at handoff time
Branch: `main`
This handoff is for a follow-on Claude/Codex agent. The user is steering a
research-grade OCR project, not a demo. The target is a PaddleOCR-centered
document OCR engine whose claims are tied to real validation and reproducible
training data, not synthetic-only benchmarks.
## Operating Rules
- Use `uv run ...` for project commands.
- Use `apply_patch` for manual edits.
- Do not commit generated corpora, rendered images, model runs, or outputs
unless explicitly requested; `data/`, `outputs/`, and `runs/` are ignored
intentionally.
- Do commit reproducibility configs, docs, tests, and source code changes.
- GPG signing fails on this machine; use:
```bash
git -c commit.gpgsign=false commit -m "<message>"
```
- Do not commit remote passwords or credentials. Hosts are documented below,
but credentials must stay in the conversation/session, not tracked files.
- Do not claim SOTA or model quality unless backed by a passed real validation
audit and benchmark.
- Synthetic OCR data is never claim evidence. It is training/ablation material.
- Validate schemas before writing converters. Do not infer fields from file
names.
## Current Research Goal
Build a PaddleOCR/PP-OCR + PP-Structure-style document engine for at least ten
languages spoken in Nepal.
Core scope:
- OCR recognition
- reading order
- tables
- figures/captions
- Markdown output
- JSON output
- optional VLM fallback for hard figures/pages
Current research angle:
> Low-resource Nepal language OCR adaptation using PDF-native weak labels,
> legacy font conversion, real Gorkhapatra validation, and script-aware
> synthetic robustness.
The real claim should be about measurable OCR/structure improvement on held-out
Gorkhapatra/Naya Nepal pages, not about "understanding everything."
## Repo State
Important recent commits:
```text
22454e2 Document cdjk language corpus migration
1f98430 Record migrated synthesis text sweep
```
The repo was checked with:
```bash
jq . configs/cdjk-language-corpus-migration-v1.json >/dev/null
git diff --check
uv run pytest tests/test_synthesis_resources.py -q
```
All passed before this handoff was written.
At handoff time, the only unrelated visible untracked path was:
```text
.playwright-mcp/
```
That is browser scratch output and should not be committed.
## Most Important Existing Docs
Read these before acting:
- `docs/ACTIVE_RESEARCH_CHECKLIST.md`
- `docs/GORKHAPATRA_LANGUAGE_PAGE_ALIGNMENT.md`
- `docs/CDJK_LANGUAGE_CORPUS_MIGRATION.md`
- `docs/SYNTHESIS_RESOURCE_AUDIT.md`
- `docs/FONT_ASSETS.md`
- `docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md`
- `docs/NEPAL_MULTILINGUAL_OCR_GOAL.md`
## What Has Been Built
### CLI Surface
The repo already has many `ocrtech` commands. Most relevant now:
```text
ocrtech audit-gorkhapatra-language-pages
ocrtech prepare-gorkhapatra-language-page-pack
ocrtech prepare-gorkhapatra-language-page-review
ocrtech audit-gorkhapatra-language-page-review
ocrtech extract-gorkhapatra-language-page-pdf-text
ocrtech convert-limbu-legacy
ocrtech audit-synthesis-resources
ocrtech prepare-limbu-limdic-text
ocrtech prepare-limbu-unicode-text
ocrtech prepare-tamang-text
ocrtech prepare-magar-text
ocrtech prepare-toolkit-parallel-text
ocrtech audit-synthesis-text-manifest
ocrtech split-synthesis-text
ocrtech render-synthesis-text-lines
ocrtech render-synthesis-text-split
ocrtech audit-rendered-synthesis-lines
ocrtech audit-rendered-degradation-split
ocrtech train-recognizer
```
### Limbu PDF-Native Extraction
Source rule for every Gorkhapatra/Naya Nepal language:
`https://gorkhapatraonline.com/publications/naya-nepal` is only the discovery
index for identifying which language/date was posted. Do not use the Naya Nepal
blog/card image as OCR, eval, or training evidence. Fetch the full paper from
the Gorkhapatra epaper date-range endpoint, for example
`https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-16&pdfdaterange2=2026-06-23&slug=gorkhapatra`,
then render the matching PDF page and extract/convert the text layer from that
same PDF page. Record PDF URL/hash/page number, rendered image hash, text
extraction provenance, and conversion map hash. Classify each language page as
`unicode_native`, `legacy_font`, `devanagari_transcription`, `latin_notation`, or
`no_text_layer`; manual transcription/native-reader work is only the fallback
when the PDF has no usable text layer or the font cannot yet be decoded.
The current successful Gorkhapatra Limbu path:
```bash
uv run ocrtech extract-gorkhapatra-language-page-pdf-text \
data/limbu-gorkhapatra-language-page-pack-2026-06-16/manifests/gorkhapatra-language-page-pack.jsonl \
--candidate-language लिम्बू \
--out data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16
```
Known result:
```text
passed=True samples=2/2 spans=776 target_spans=304 fonts=10 warnings=0
sample_id=gorkhapatra-language-page-0001-01
pdf_page_number=6
target_font=NamdhinggoSILL
target_spans=304
embedded_font=UFNWLJ+NamdhinggoSILL
```
Page conclusion:
- page 6 is the Limbu/Sirijonga page
- page 7 is the same-date non-Limbu page
The PDF text layer exposes legacy ASCII-like Namdhinggo text, not Unicode
Limbu. Example:
```text
legacy: kfMG g' ;fK;SF[ yf]af]cf
unicode: ᤐᤠ᤺ᤴ ᤏᤢ ᤛᤠᤵᤛᤰᤖᤧ ᤌᤥᤒᤥᤀᤠ
```
The converter is implemented in `src/ocrtech/limbu_legacy.py` using the vendored
SIL map:
```text
data/mappings/limbu-legacy/Limbu.map
```
### Limbu Review Dashboard
Generated dashboard path:
```text
data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html
```
Correct local file URL:
```text
file:///Users/cdjk/github/llm/ocr-tech/data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html
```
This is the correct browser UI for the Gorkhapatra PDF-native Limbu page. It
shows the full rendered PDF image with bounded boxes; clicking a box selects the
matching right-side row with converted text and the corresponding rendered span
crop. Do not substitute the PDF-native verification-bundle HTML for this task;
that was the wrong/broken UI.
Local server used:
```bash
uv run python -m http.server 8765 --bind 127.0.0.1
```
Dashboard URL:
```text
http://127.0.0.1:8765/data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html
```
Important bug already fixed in the generated dashboard: clicking a bounding box
must not re-render or scroll the image/page back to the top. The fix removed
automatic `scrollIntoView` from bounding-box selection.
If this dashboard is regenerated by code later, preserve this behavior:
```js
function selectSpan(id){
selected=id;
document.querySelectorAll('.box.selected,.span.selected').forEach(el => el.classList.remove('selected'));
document.querySelectorAll(`[data-span="${id}"],[data-row="${id}"]`).forEach(el => el.classList.add('selected'));
}
```
### Migrated Corpus
Source host:
```text
cdjk@100.110.148.43
machine: cdjks-MacBook-Pro.local
external volume: /Volumes/TRANSCEND
```
Do not write generated data back to that host. Treat it as a source for corpus
and dictionaries unless the user explicitly asks otherwise.
Tracked migration policy:
```text
configs/cdjk-language-corpus-migration-v1.json
```
Ignored local copy:
```text
data/external-language-resources/cdjk-100.110.148.43/
```
Copied roots:
```text
dhimal-toolkit
gurung-language-toolkit
limbu-speech-toolkit
ltk
magar-toolkit
rai-speech-toolkit
```
Migration result:
```text
2307 text/resource files transferred
1.6G local size
resource audit: passed=True roots=6/6 files=2308 warnings=0
```
### Prepared Synthesis Text
Generated synthesis text is ignored under `data/synthesis-text/`, but the
documented state is:
```text
20 manifests
228212 total samples
45258 duplicate rows removed/reported by converters
202560 Devanagari rows
10337 Mixed_Devanagari_Latin rows
6521 Latin rows
8772 true Unicode Limbu rows
14 Mixed_Limbu_Devanagari rows
3 Mixed_Limbu_Latin rows
5 Other rows
```
Important generated manifests:
```text
data/synthesis-text/cdjk-limbu-limdic-2026-06-16/limbu-limdic-text.jsonl
data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered/limbu-only.jsonl
data/synthesis-text/cdjk-tamang-2026-06-16/tamang-text.jsonl
data/synthesis-text/cdjk-magar-2026-06-16/magar-text.jsonl
data/synthesis-text/cdjk-gurung-2026-06-16/toolkit-parallel-text.jsonl
data/synthesis-text/cdjk-dhimal-parallel-2026-06-16/toolkit-parallel-text.jsonl
data/synthesis-text/cdjk-dhimal-mono-2026-06-16/toolkit-parallel-text.jsonl
data/synthesis-text/cdjk-magar-parallel-extra-2026-06-16/toolkit-parallel-text.jsonl
data/synthesis-text/cdjk-magar-mono-extra-2026-06-16/toolkit-parallel-text.jsonl
data/synthesis-text/cdjk-rai-*-2026-06-16/toolkit-parallel-text.jsonl
```
All synthesis text audits passed with `claim_eligible=0`.
Important interpretation:
- True Unicode Limbu exists from Limdic and should render with Namdhinggo after
filtering `script == Limbu`.
- Most Gurung, Dhimal, Magar, Tamang, and Rai rows are Devanagari-transcribed
language text, not native-script OCR.
- Bantawa, Khaling, Koyu, and Thulung include substantial Latin phonetic data.
These are useful for language resources but should not be called native-script
OCR data.
### Synthetic Rendering
Corrected Limbu-only smoke render:
```bash
uv run ocrtech render-synthesis-text-lines \
data/synthesis-text/cdjk-limbu-limdic-2026-06-16/filtered/limbu-only.jsonl \
--out data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only \
--font-path data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-Regular.ttf \
--limit 120 \
--degradation-profile clean \
--degradation-profile scan \
--degradation-profile low_light \
--degradation-profile uneven_light \
--degradation-profile camera_left \
--degradation-profile phone_photo
```
Audit:
```bash
uv run ocrtech audit-rendered-synthesis-lines \
data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only/manifests/rendered-line-manifest.jsonl \
--label-file data/rendered-synthesis/cdjk-limbu-smoke-2026-06-16-limbu-only/labels/train.txt \
--out outputs/cdjk-limbu-rendered-smoke-audit-2026-06-16-limbu-only
```
Result:
```text
passed=True samples=120 claim_eligible=0 issues=0 warnings=0
degradation_counts:
camera_left: 20
clean: 20
low_light: 20
phone_photo: 20
scan: 20
uneven_light: 20
```
Important lesson: an earlier mixed-script render produced tofu boxes because
Devanagari/Latin rows were rendered with a Limbu font. Always filter by script
before choosing a font.
## Current Strategic Decision
The user asked what next. The right order is:
1. Build Gorkhapatra validation for all target languages.
2. Create language/script/font matrix from actual Gorkhapatra PDF evidence.
3. Generate split-safe synthetic data per script/font bucket.
4. Research and execute PaddleOCR recognizer fine-tuning.
5. Keep PP-Structure/layout mostly stock until recognizer evidence says
detection/layout fine-tuning is required.
Do not scale synthetic data first without real validation. It will create a
large training set but no credible paper claim.
## Immediate Next Task
Build a reusable all-language Gorkhapatra validation generator.
It should turn this manual Limbu success into a repeatable command/workflow:
```text
Naya Nepal publication card
-> date/language/article metadata
-> matching epaper PDF
-> candidate language-page pages
-> rendered page image
-> PDF-native spans/fonts/bboxes/codepoints
-> optional legacy conversion by script/font
-> review dashboard
-> reviewed JSON/CSV annotations
```
Suggested CLI shape:
```text
ocrtech prepare-gorkhapatra-language-validation
```
Likely arguments:
```text
--language-label <Nepali label>
--publication-pages <N>
--epaper-url <date-range endpoint>
--out <data/gorkhapatra-validation-...>
--target-font <optional font name>
--legacy-converter <optional converter id, e.g. limbu-namdhinggo>
--render-pages
--download-pdfs
```
Output should include:
```text
validation.json
validation.md
samples/*.pdf-text.json
samples/*.pdf-text.txt
spans.csv
fonts/
page-images/
span-renders/
dashboard.html
review.csv
review.jsonl
```
Dashboard requirements:
- Full page image with clickable bbox overlay.
- Right panel with selected span, PDF-rendered crop, raw extracted text,
converted Unicode text when available, font, bbox, codepoints.
- Review status controls:
- `accept`
- `needs_fix`
- `bad_segment`
- `drop`
- `wrong_page`
- `wrong_language`
- Must preserve image/page scroll when clicking bbox.
- Must not ask the user to transcribe text if PDF-native extraction/conversion
already produces usable Unicode text.
- Must save/export annotations as JSONL/CSV.
## Language Validation Targets
At least ten target languages are desired. Current likely sources:
- Limbu/Sirijonga
- Tamang
- Magar
- Gurung
- Dhimal
- Newar/Newa
- Bantawa/Rai
- Chamling/Rai
- Kulung/Rai
- Athpariya/Rai
- Khaling/Rai
- Yakkha/Rai
- Sunwar/Rai
- Lohorung/Rai
- Dumi/Rai
- Thulung/Rai
- Koyu/Rai
But Gorkhapatra validation must decide actual script form:
- Unicode native script
- legacy font encoding
- Devanagari transcription
- Latin notation
- unsupported/no text layer
Do not assume native Unicode just because the language has a script.
## Training Research Direction
Use primary sources before changing training code. Current understanding:
- PaddleOCR/PP-OCR remains the primary engine family.
- Start by fine-tuning recognizer, not layout/detection.
- Use PP-Structure/PP-StructureV3-style parsing for document layout and
structured output.
- Consider PaddleOCR-VL only as fallback/teacher/comparator, not the default
small local engine.
Recognizer-first rationale:
- The repo already has line/synthesis text and rendered line labels.
- Recognition turns cropped text images into text.
- Detection/layout claims require bbox/layout ground truth, which is not yet
mature across languages.
- Fine-tuning detection before measuring detection failures is premature.
Training prerequisites before a serious run:
- Split synthesis text with `split-synthesis-text`.
- Exclude real validation/reference text hashes from synthetic training.
- Render train/eval with font readiness and degradation audits.
- Keep English/Nepali non-regression sets.
- Package PaddleOCR recognizer training recipe.
- Evaluate on Gorkhapatra real pages and held-out synthetic rows.
## Remote Machines
Known hosts from the session:
```text
k2@100.97.238.103
cdjk@100.110.148.43
cdjk@100.117.21.47
```
Do not commit passwords. The user provided credentials in chat previously, but
tracked docs must not contain them.
Important storage findings:
- `cdjk@100.110.148.43` has `/Volumes/TRANSCEND`, about `3.6Ti` total and
`2.0Ti` available at inventory time. Use as source for corpus/dictionary
resources unless instructed otherwise.
- `cdjk@100.117.21.47` root was full at inspection time:
```text
/ 915G 865G 3.7G 100%
/mnt/tts_cache_nvme 234G 150G 72G 68%
```
Do not dump large generated data to a nearly-full root filesystem.
## Source Files To Inspect Before Editing
For Gorkhapatra flow:
```text
src/ocrtech/source_audit.py
src/ocrtech/cli.py
src/ocrtech/limbu_legacy.py
tests/test_source_audit.py
tests/test_limbu_legacy.py
```
For synthesis:
```text
src/ocrtech/synthesis_resources.py
tests/test_synthesis_resources.py
configs/cdjk-language-corpus-migration-v1.json
```
For fonts/language registry:
```text
src/ocrtech/font_assets.py
src/ocrtech/language_registry.py
configs/namdhinggo-font-assets-v1.json
configs/nepal-font-assets-v1.json
languages/nepal-v1.json
```
For training:
```text
src/ocrtech/training.py
src/ocrtech/models.py
docs/MODELS.md
docs/MULTILINGUAL_PADDLEOCR_RESEARCH_DIRECTION.md
```
## Known Pitfalls
- The user is frustrated by fake OCR candidates and manual busywork. Do not ask
them to validate bad OCR text when PDF-native text extraction can provide the
answer.
- For Limbu, PDF text was extractable but legacy encoded. The correct path was
PDF-native text -> Namdhinggo legacy conversion -> Unicode display.
- Dashboard bbox clicks must not reset scroll.
- Do not treat Devanagari-transcribed language data as native script OCR.
- Do not render mixed-script rows with a single script-specific font.
- Do not collapse mixed schemas into one converter command by dropping
validation. Split sources by schema instead.
- Do not publish synthetic-only claims.
## Recommended Next Commit
Implement the reusable validation generator and tests:
1. Refactor current generated dashboard logic into source code.
2. Add a CLI command that accepts a manifest/PDF-text extraction dir and emits
dashboard + review CSV/JSONL.
3. Preserve current Limbu dashboard features:
- full page image
- bbox overlay
- PDF-rendered span crop
- converted Unicode text
- raw legacy text
- font metadata
- stable scroll behavior
4. Add fixture tests for dashboard generation and annotation schema.
5. Document command in `docs/GORKHAPATRA_LANGUAGE_PAGE_ALIGNMENT.md`.
6. Run focused tests:
```bash
uv run pytest tests/test_source_audit.py tests/test_limbu_legacy.py -q
git diff --check
```
Then commit with GPG signing disabled and push.