Download ocr-tech-mps-integration/RESEARCH.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 23.8 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/RESEARCH.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech-mps-integration/RESEARCH.md
-
curl -L -o RESEARCH.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/RESEARCH.md
Research State: Nepal Language Pack OCR
Updated: 2026-08-09
Current Goal
The end goal is an OCR system covering the multiple languages of the Nepal pack, built out of Paddle, with Limbu and Newar (Nepal Bhasa) as the first two focus languages (user steer, 2026-07-04). The system should accept real book/page images or PDFs for any language in the pack and output editable Unicode text with page, line, crop, confidence, and provenance evidence preserved. Limbu stays the proven template — it has the strongest recognizer and the deepest harvest. Newar is the deliberate second: it has its own native script (Prachalit), real cultural demand, and is currently the weakest trained recognizer, so it exercises every part of the method that Limbu already proved. Every stage that was built Limbu-specific must generalize: harvest, silver-label extraction, recognizer training, script routing, page/book OCR, and eval. The product is the full pipeline: page detection, script-aware routing to per-script recognizers, target-region and furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and review artifacts for hard lines. YOLO line detection stays a diagnostic/probe; Paddle-first is the production story.
The Language Pack
Two inventories exist and they are not the same thing:
- The registry
languages/nepal-v1.jsontracks 18 target languages (17 Nepal + English) with scripts, fonts, dictionaries, and corpus status. - The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found
50 normalized languages (53 raw labels) in 400 articles over ~7 months (see the
harvest section). Labels include all registry languages plus Chamling Rai, Bahing,
Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo,
Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi,
Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more.
Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा,
ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) —
tools/build_gorkhapatra_language_pack_inventory.pyowns that normalization and the label→registry mapping.
Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded
2026-06-18 scorecard in docs/RECOGNIZER_GOAL_PROGRESS.md):
- Limbu/Sirijonga 0.041 real-print (champion:
runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23, cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval). - 8 Devanagari languages 0.075–0.129 real-print (Majhi, Magar, Sunuwar-Deva, Magahi, Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer.
- Sunuwar (Jenticha) 0.121 real-crop after the same-font retrain (0.342 synth-only).
- Tirhuta 0.343 and Newa 0.613 synthetic-only — weak, transliteration-fed, honest fails; Newa needs real printed-register Nepal Lipi, not more pixels.
- Kirat Rai (70 real crops) and Lepcha (132 real crops) blocked on legacy-font converter coverage outside their eval editions; Gurung Khema has no real print anywhere (revival script) — synthetic-only by nature, claim-scoped accordingly.
The near-term pack claim should therefore be script-family shaped: one Sirijonga recognizer, one shared Devanagari recognizer covering the many Devanagari-written languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa, Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval crops exist.
Focus order (2026-07-04 steer): Limbu first, Newar second. Newar state: the
Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own
synthetic eval; transliteration-fed training failed; real printed-register Nepal
Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar
route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20
Gorkhapatra pages; newar added to the registry (Devanagari primary + Prachalit
secondary; audit passes); a frozen by-date pack built at
outputs/newar-devanagari-silver-pack-v1/ (train 4,096 / dev 1,528 / test 1,139
lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently
verified; pack sha256
1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85); and the
first real-print Devanagari-Newar baseline was scored locally (CPU export
route, tools/score_recognizer_pack.py, per docs/LOCAL_RECOGNIZER_SCORING.md):
the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at
codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158 (n=1,139 silver
lines — machine-converted labels, diagnostic, not claim-grade). The same-day
warm-start fine-tune (newar-ft-v1: deva-v2 base, LR 2e-5 cosine, 60 epochs,
batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance
ashish-experiments, ~68 min; artifacts + train log at
outputs/aws-newar-deva-ft-v1/) scores the same frozen test at codepoint CER
0.0465, grapheme CER 0.0955, exact-line 0.245 — a 25/27% relative CER drop and
+55% exact-line over the deva-v2 baseline, scored through the identical local
CPU path. Newar's grapheme CER now sits inside deva-v2's own-language range
(0.075–0.129): the second focus language reached Devanagari-pack parity from
zero in one day. Same claim boundary: silver machine-converted labels,
diagnostic, not claim-grade until native review. Next Newar lever: native
Prachalit (real printed-register data), and page-level Newar eval for the
region-filtering track.
Source Rule
For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata
only. Use https://gorkhapatraonline.com/publications/naya-nepal to find the language,
date, and article, but do not use the blog/card image for OCR, training, or evaluation
when the full paper exists. The authoritative source is the matching Gorkhapatra epaper
PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus
the text layer extracted from that same PDF page, with PDF URL/hash, page number,
rendered image hash, extraction command/version, font evidence, and converter/map hash
recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with
src/ocrtech/limbu_legacy.py + data/mappings/limbu-legacy/Limbu.map. The same
pattern — classify the text layer, then build an explicit converter only when the font
requires it — replicates to every other language. Manual transcription is the fallback
for no usable text layer, not the default.
Getting More Gorkhapatra Pages (the specified harvest recipe)
All numbers below were verified live on 2026-07-04.
Archive size. The Naya Nepal publication archive paginates at
https://gorkhapatraonline.com/publications/naya-nepal?page=N with 5 articles per
page and runs to page 352 before drying up: **1,750 language-page articles total**
(about 2 per day going back years). The 60-page crawl already yielded 300 items across
53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly
older dates.
Epaper endpoint behavior (the critical constraint). The date-range endpoint
https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra
caps results at 30 PDFs per query and ignores page=N pagination. A wide window
silently drops most dates — the 2026-07-04 six-window discovery pass aligned only
111/300 items for exactly this reason, with unmatched dates interleaved among matched
ones. Seven-day windows return complete daily coverage (7/7 verified). Always
harvest with weekly windows.
The recipe (one command; add one --epaper-url per week to cover the publication
date span being harvested):
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
ocrtech audit-gorkhapatra-language-pages \
--out outputs/gorkhapatra-language-pages-full-<date> \
--publication-url https://gorkhapatraonline.com/publications/naya-nepal \
--publication-pages 80 \
--epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \
--epaper-url '...one per week, going back...' \
--download-pdfs \
--render-pages \
--timeout-seconds 180
The command aligns every publication card (all languages at once — alignment is
language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL),
searches pages for the language-page markers (भाषा पृष्ठ and its legacy-encoding
form efiff k[i7), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly
windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with
400 publication items, 195 aligned dates, 394 rendered language-page hits, zero
warnings at outputs/gorkhapatra-language-pages-full-2026-07-04/. Batch 2
(2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more
items: 798 items, 294 aligned dates, 589 further page hits, at
outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/. The merged normalized
inventory (tools/build_gorkhapatra_language_pack_inventory.py) shows 56
languages, 983 rendered language-page hits, most languages with ~20–27 pages
spanning 14 months, at outputs/language-pack-inventory-merged-2026-07-04/. For
comparison, the entire previous state of the project had 8 usable pages for one
language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least
March 2024 — verified the epaper endpoint serves complete weeks there) is another
~2× and stays month-batchable with the same recipe.
Scaling to the full archive. Full depth is ~700+ unique dates ≈ 35 GB of PDFs.
Do not do it in one run: batch by month (4–5 weekly windows + the matching
publication-page depth), run extract-gorkhapatra-language-page-pdf-text per batch,
then keep rendered pages, extracted text, and hashes; PDFs can be pruned after
extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume
(cdjk@100.110.148.43:/Volumes/TRANSCEND) is the right home for a full-depth mirror.
Per-language downstream. After harvest, run per language:
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
ocrtech extract-gorkhapatra-language-page-pdf-text \
<pack-manifest.jsonl> \
--candidate-language <label> \
--target-font <font> \
--out outputs/<language>-pdf-text-<date>
then classify each page's text layer (native Unicode / legacy font / Devanagari transcription / Latin / none) and only then decide converter vs manual transcription.
Font/text-layer facts already established (30-page, 15-language extraction at
outputs/gorkhapatra-multilingual-language-pages-current-2026-06-24-pdf-text/):
- The body font of non-Limbu language pages is
nayanepal— a legacy 8-bit Preeti-family encoding, not Unicode (8,845 of 11,791 spans; the text layer readsk[i7 ;+of]hswhere the page shows पृष्ठ संयोजक). The converter already exists:nepal_ttf2utf.convert_devanagari(font="nayanepal")in the sibling repo/Users/cdjk/github/llm/nepal-ttf2utf(Preeti base map + nayanepal extension glyphs + NFC, ~97–99% clean on real Gorkhapatra pages), already used bytools/build_gorkhapatra_devanagari_real.pyfor the 79k-row real Devanagari corpus. The remaining leverage is wiring it per language: run it over each harvested language page and emit per-language silver line packs with the same faithfulness gate (clean-Unicode allow-set,conversion_gapaccounting, leakage checks) that the Devanagari pack builder already implements. The same sibling repo also hosts the limbu, kiratrai, lepcha, and sunuwar legacy converters. - Tibetan-script fonts (
TibetanMachine,MonlamUniOuChan5,Qomolangma-*) appear on Mugal and related pages, but every such span has empty extracted text (verified 2026-07-04 across all 2026-06-24 samples) — Tibetan-script pages are visual-only, like the known Limbu2082-11-12date. They need OCR bootstrap or manual transcription; no converter is possible. Gorkhapatra-*,Nagarik-*,TimesNewRoman*,MyriadPro-*are masthead/furniture fonts — useful as non-target filters for target-region selection.NamdhinggoSILLappears only on Limbu dates (the existing Limbu path).
What We Built (Limbu template, still true)
The Gorkhapatra alignment path: publication-page discovery, epaper-date matching, PDF
download/rendering, page-marker search, PDF-native span/font extraction, review
sheets, reference templates, and dashboard visualization. The expanded Limbu harvest
has 9 discovered dates, 8 usable PDF-text-layer pages, 2,441 line rows, 2,430 usable
silver labels (outputs/limbu-gorkhapatra-pdf-backed-silver-2026-07-03-v1/).
The Limbu recognizer training/eval path: 48,507-row combined pack; real-heavy fine-tune champion at codepoint CER 0.011336 / grapheme 0.024736 / exact-line 0.729 on the frozen 255-line silver eval. Synthetic hardening lessons: targeted+Unicode packs regressed; v3 dictionary warm-start catastrophically failed (~0.98 CER); keep the real-heavy v2 champion; future gains come from real/book-domain harvesting or a deliberate rehead, not blind synth rows.
The page/book OCR loop: local + HF Space (spaces/limbu-ocr/, deployed private at
https://huggingface.co/spaces/ampixa/limbu-ocr) with line OCR, Paddle Page OCR,
Paddle Book OCR, image-line fallback, runtime checks, annotated pages, and crops.
Page-level diagnostics on the 8-page silver pack: Rows LTR + y/conf70 filter scores page cp CER 0.414330, line F1 0.963009, matched-line cp CER 0.026039, order accuracy 0.993176. Recognition on matched lines is strong; full-page CER is dominated by page reconstruction, target-region selection, and furniture filtering — that is the pipeline problem to solve, and it is language-independent.
Supervised line-detector infrastructure from CDC PDF geometry (287 pages, 26,160 annotations, mAP50 ≈ 0.921 on PDF-rendered val) — evidence the detector path works, not a solved product.
A claim-grade eval harness (versioned packs, hashes, provenance, normalization profiles, cp/grapheme CER, CIs, paired comparisons, leakage manifests) — parked as a guardrail; do not let it eat work blocks that should go to the pipeline.
Current Working State
The strongest component is the Sirijonga line recognizer. The weakest component is
end-to-end page/book reconstruction: target-region selection, furniture filtering,
line preservation, and ordering. Of the three Limbu-locked code spots, two were
generalized on 2026-07-04 (Codex worker, tests green): _script_ratio in
src/ocrtech/engines.py is now driven by the shared SCRIPT_CODEPOINT_RANGES table
covering all pack scripts (devanagari, limbu/sirijonga, latin, sunuwar/jenticha,
lepcha, kirat_rai, newa/prachalit, tirhuta, gurung_khema, ol_chiki, tibetan, with
aliases), and the image-line filter in src/ocrtech/image_lines.py accepts any
script key from that same table. The third spot fell the same day: the
src/ocrtech/limbu_runtime.py preflight now takes required_script (default
limbu, validated against the shared table; CLI --required-script), so a router
card for any pack script can be preflighted. The engine classes themselves (CandidateEngine, PaddleOcrEngine,
ScriptSelectCompositeEngine, build_image_line_document, structure.py reading
order) are already parameterized — the rest of the generalization is registry-driven
wiring, not a rewrite.
Apple MPS Recognizer Training Evidence (2026-08-09)
Stock PaddlePaddle is not the immediate Apple-GPU route. A direct PaddlePaddle
3.3.1 probe on k2 exposed no custom devices and rejected mps, metal, and
custom_device:metal. Paddle's official PaddleCustomDevice MPS backend exists,
but is currently a version-0.0.1 prototype with too small an operator surface to
establish PP-OCRv5 training.
The practical path is an Apache-2.0 vendored subset of
frotms/PaddleOCR2Pytorch pinned at
c05307fbd61e575d55f0fef0334026c7536aec76: PPLCNetV3 scale 0.95 plus the
SVTR/CTC branch of the PP-OCRv5 mobile recognizer. Dynamic pooling preserves the
recognition width (T = input_width / 4), and a localized contiguity fix before
the SVTR encoder's conv3 makes the model safe for the tested MPS runtimes. The
model and log-softmax run on MPS; CTCLoss runs on CPU and autograd returns
finite gradients to MPS for the optimizer update. The NRTR auxiliary branch is
frozen and not part of this CTC loss, so this is not a complete reproduction of
PaddleOCR's multi-loss training recipe.
One-step evidence is green on both Macs. The end-to-end mbp2 run used Python
3.11.15 and Torch 2.13.0: [4,3,48,160] -> [4,40,7], loss 29.883989, 582 finite
MPS gradient tensors, and a non-zero optimizer update across synthetic Latin,
Devanagari, and Limbu samples. Direct width-320 forward/backward/update checks
also passed on mbp2 and on k2 with Torch 2.12. These cold one-step timings
are diagnostics, not benchmarks.
The checkpoint boundary is strict: this backend writes PyTorch checkpoints,
not Paddle .pdparams; conversion requires explicit tensor mapping and frozen
inference-parity tests. mbp2 and k2 should run independent experiments with
distinct seeds and run directories, not multi-host MPS DDP. The remaining proof
is a multi-epoch real-manifest run with resume/reload, memory observation, and
canonical held-out CER. See docs/PADDLEOCR_MPS.md for exact smoke artifacts,
hashes, compatibility limits, and host guidance.
The Paddle Pipeline (whole pipeline, not just recognizers)
Target architecture, all stages Paddle-first and language-pack-parameterized by the
per-language pipeline config configs/nepal-language-pack-pipeline-v1.json
(50 languages: label, registry id, scripts, recognizer artifact, dictionary,
converter, text-layer class, harvest counts — generated 2026-07-04 from the
inventory):
- Capture/prep — existing
prepare-limbu-capturepath, already language-agnostic in behavior; rename/rescope. - Detection — Paddle text detection as production default; YOLO line detector and morphology image-line boxes stay as diagnostics/rescue passes.
- Script identification & routing — generalize
_script_ratioto a Unicode-block-driven scorer over the pack's script list (Devanagari, Limbu, Jenticha/Sunuwar, Lepcha, Kirat Rai, Newa, Tirhuta, Gurung Khema, Ol Chiki, Tibetan, Latin), then route each line to the script's recognizer from the registry. - Recognition — per-script PP-OCRv5 recognizers (existing runs) with per-script dictionaries and input-width profiles (Sirijonga width-480 lesson).
- Target-region & furniture filtering — non-oracle region selection using layout plus font/language signals; furniture fonts and masthead geometry as negative evidence. This is the main open research item; score against the Rows LTR + y/conf70 baseline.
- Reading order & assembly —
auto_layoutselection (wide-column RTL vs rows) per page, conservative assembly stays diagnostic. - Export — Unicode text, JSON with provenance, Markdown; review queues + dashboards per page (existing Limbu artifact shapes, renamed pack-wide).
- Eval guardrail — frozen real-page packs per language: page CER/WER, line P/R/F1, reading order, blank/extra/missed lines, review burden.
PP-StructureV3 and PaddleOCR-VL remain the documented fallback/teacher direction for
hard layouts, per docs/NEPAL_MULTILINGUAL_OCR_GOAL.md.
Next Work
Finish the 2026-07-04 full harvest and inventoryDone 2026-07-04: 394 language-page hits, 50 languages, inventory manifests atoutputs/language-pack-inventory-full-2026-07-04/.Per-language silver packs via the existingDone 2026-07-04 (nayanepalconvertertools/build_language_pack_silver_line_packs.py, tests green): the batch-1 run produced 116,287 kept silver lines across 50 languages, zero errors, with per-line provenance (PDF hash, page, bbox, raw legacy bytes, converted text, image hash) atoutputs/language-pack-silver-lines-2026-07-04/; spot-checked conversions read as clean, language-plausible Devanagari. The batch-2 run over the older 589 page hits kept 183,324 more lines (49 languages, zero errors) atoutputs/language-pack-silver-lines-batch2-2026-07-04/— 299,611 per-language-attributed silver lines total across the two batches (~2.4 GB of crops). Run the tool withuv run --python 3.11 --with pymupdf --with npttf2utf—nepal_ttf2utfneeds thenpttf2utfbase-map package at runtime, and without it every span fails (cleanly, with per-span error records, but kept=0). Honest caveat: lines are page-attributed, not language-verified — a language page also carries Nepali furniture/boxes, and pages whose body is not nayanepal Devanagari (Urdu, the Tibetan-script pages) yield mostly furniture lines. A language-ID/lexicon filter before per-language recognizer training is the follow-up gate.- Newar track (second focus language): add the Newar registry entry (Devanagari + Prachalit rows); build the Devanagari-written Newar recognizer eval/train split from the 6,800 harvested silver lines (by-date split, frozen test); keep native Prachalit as the separate longer arc — real printed-register Nepal Lipi harvesting, not more transliteration, is the lever.
- Classify Tibetan-script pages (Hyolmo/Mugal/Bhote) and decide converter vs native Unicode.
Generalize the pipeline wiringDone 2026-07-04: all three Limbu-locked spots generalized (shared script table, filter keys,--required-scriptpreflight), tests green;configs/nepal-language-pack-pipeline-v1.jsonholds the per-language wiring.- Target-region/furniture filtering stage for Paddle page/book OCR, scored against the current baseline on the 8-page Limbu pack and at least one Devanagari-language pack (Newar is the natural candidate given the focus order).
- Only after page reconstruction improves, export high-quality real/silver line crops back into recognizer training, per language.
- Month-batch the archive back-harvest toward the full ~1,750-article depth, storing the mirror on the remote 2 TB volume.
Mistakes To Avoid
Do not use Naya Nepal blog images as training/eval evidence when the PDF exists. Do
not use wide epaper date windows — the endpoint silently caps at 30 PDFs per query;
weekly windows only. Do not treat unnormalized Gorkhapatra labels as distinct
languages (थारु vs थारू). Do not treat the nayanepal text layer as Unicode — it is
legacy Preeti-family and needs the converter before any silver label is real. Do not
report Paddle norm_edit_dis as CER; real CER comes from tools/eval_metrics.py
micro CER over canonical predictions. Do not treat synthetic data as claim evidence,
and label Gurung Khema results as synthetic-by-nature. Do not promote the
targeted+Unicode checkpoint or v3 dictionary warm-starts. Do not spend long blocks on
generic eval-harness work unless it directly improves the book/page OCR loop. Do not
frame video-to-page work as OCR scope; reduce video to stills first. Do not present
page OCR as solved: matched-line recognition is strong, but full-page CER is still
dominated by reconstruction and region selection.