ocr-projects-archive / ocr-tech /docs /ACTIVE_RESEARCH_CHECKLIST.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
22.4 kB

Active Research Checklist

This checklist records the current steering decision for the Nepal multilingual OCR paper track. Keep it updated when remote data, synthesis converters, validation packs, or claim gates change.

Current Goal

Build a PaddleOCR-centered document OCR engine for at least ten languages spoken in Nepal. Limbu/Sirijonga is the first non-Devanagari target. The engine must support OCR, reading order, tables, figures, captions, Markdown, and structured JSON. Synthetic data can train and ablate the system, but paper claims require real held-out validation, especially Gorkhapatra/Naya Nepal page evidence.

Non-Negotiable Gates

  • Keep PaddleOCR/PP-OCR as the primary engine family.
  • Use PP-Structure-style document extraction for layout, reading order, table reconstruction, figures, captions, Markdown, and JSON.
  • Use PaddleOCR-VL or another VLM only as fallback, teacher, or hard-page comparator; report fallback rate separately.
  • Do not call a language synthesis-ready until a text manifest passes ocrtech audit-synthesis-text-manifest.
  • Do not call rendered OCR data training-ready until rendered outputs pass ocrtech audit-rendered-synthesis-lines.
  • For split synthesis text, render with render-synthesis-text-split so train/eval labels and rendered audits are created together.
  • Do not use synthetic samples as claim evidence.
  • Do not promote Gorkhapatra/Naya Nepal pages to claim evidence until the review audit and reference audit pass.
  • Keep Nepali and English non-regression gates in every claim-grade run.

cdjk Remote Synthesis Path

Remote host verified read-only on 2026-06-16:

host: cdjk@100.110.148.43
machine: cdjks-MacBook-Pro.local
disk: /Volumes/TRANSCEND, 3.6Ti total, 2.0Ti available

Confirmed ~/github/llm leads:

  • /Users/cdjk/github/llm/limbu-speech-toolkit
  • /Users/cdjk/github/llm/rai-speech-toolkit
  • /Users/cdjk/github/llm/dhimal-toolkit
  • /Users/cdjk/github/llm/gurung-language-toolkit
  • /Users/cdjk/github/llm/magar-toolkit
  • /Users/cdjk/github/llm/ltk
  • /Users/cdjk/github/llm/ltk/limbu
  • /Users/cdjk/github/llm/ltk/tamang
  • /Users/cdjk/github/llm/ltk/magar
  • /Users/cdjk/github/llm/nepa-newa-text-frontend
  • /Users/cdjk/github/llm/nepali_text_frontend
  • /Users/cdjk/github/llm/devanagari-OCR
  • /Users/cdjk/github/llm/nllb-better

Confirmed /Volumes/TRANSCEND leads:

  • /Volumes/TRANSCEND/bible_brain_nepal/lif/LIFWBT_Limbu
  • /Volumes/TRANSCEND/bible_brain_nepal/dhi/DHIDPI_Dhimal
  • /Volumes/TRANSCEND/bible_brain_nepal/gvr/GVRWBT_Gurung
  • /Volumes/TRANSCEND/bible_brain_nepal/new/NEWNCL_Newari
  • /Volumes/TRANSCEND/bible_brain_nepal/taj/TAJWBT_Tamang__Eastern
  • /Volumes/TRANSCEND/bible_brain_nepal/npi/NPIBIB_Nepali
  • /Volumes/TRANSCEND/bible_brain_nepal/npi/NPIDPI_Nepali
  • /Volumes/TRANSCEND/nepali_dataset_prep
  • /Volumes/TRANSCEND/noiseremover_nepal18_timed_pipeline/bible_brain
  • /Volumes/TRANSCEND/noiseremover_nepal18_timed_pipeline/speaker_cluster_limbu

Next remote steps:

  • Selectively migrate text corpus/dictionary resources from cdjk@100.110.148.43 into ignored local data/ without writing generated corpus data back to the source machine. Tracked policy: configs/cdjk-language-corpus-migration-v1.json.
  • Run ocrtech audit-synthesis-resources on the migrated roots: passed=True roots=6/6 files=2308 warnings=0.
  • Prepare and audit first migrated synthesis-text manifests: Limbu Limdic 26152 rows, Limbu-only subset 8772 rows, Tamang 46314 rows, and Magar Western dictionary 1182 rows; all audits passed with claim_eligible=0.
  • Render a corrected Limbu-only Namdhinggo smoke set with six degradation profiles: 120 images, rendered audit passed=True, issues=0, warnings=0.
  • Convert the same local migration into full split train/eval synthesis manifests with exclusion sets and license policy before large-scale rendering.
  • Inspect actual schemas for each candidate file before writing converters. Do not infer fields from filenames.
  • Run explicit field-mapped conversion recipes for Gurung, Dhimal, and Rai dialect corpora discovered in the migration.
  • Run explicit field-mapped conversion for extra Magar mgp/source/mscore monolingual rows and mgp/npi lexicon/parallel rows.
  • Audit 20 migrated synthesis-text manifests after the second sweep: 228212 total rows, 202560 Devanagari rows, 8772 true Unicode Limbu rows, and all manifest audits passed with claim_eligible=0.
  • Add a combined index/report command for migrated synthesis-text manifests so the aggregate counts do not depend on ad hoc jq commands.
  • Add an explicit-field converter for remote toolkit JSONL/TSV/CSV parallel corpora.
  • Add audit-synthesis-text-promotion so remote synthesis text must carry split policy and must not overlap excluded eval/reference text before rendering.
  • Add split-synthesis-text so promoted train/eval synthesis text is grouped by text_sha256, stamped with split policy, and promotion-audited.
  • Smoke-run the converter on real remote Magar toolkit JSONL and pass audit-synthesis-text-manifest with Magar/Nepali Devanagari gates.
  • Smoke-run that converter on selected Limbu, Rai, Dhimal, Gurung, and Magar remote toolkit files, then audit the emitted synthesis-text manifests.
  • Promote remote toolkit smoke commands into full split synthesis-text manifests that pass split-synthesis-text with exclusion sets and license policy appropriate to the run.
  • Prioritize remaining converters in this order: Limbu Pangloss XML/TSV, Bible Brain text manifests, Nepali prep TSV/transcript roots.
  • Record license state per source. Bible Brain and derived text/audio roots stay pending_review until cleared.
  • Emit source path, source SHA-256, text SHA-256, language, script, Unicode block counts, split policy, and claim_evidence_eligible=false for every synthesis text row.
  • Deduplicate synthesis text against all real validation references before rendering.
  • Render promoted train/eval synthesis text with verified fonts and --require-font before using labels for PaddleOCR training.
  • Verify fonts and Unicode renderability before rendering new scripts.
  • Add Namdhinggo 3.100 as the primary Limbu/Sirijonga font candidate and keep Noto Sans Limbu as a fallback runtime font.
  • Add configs/namdhinggo-font-assets-v1.json so Namdhinggo 3.100 is reproducibly provisioned from the upstream SIL release ZIP with package and extracted-member SHA-256 checks.
  • Add ocrtech inventory-fonts for directory/ZIP font collection audits with SHA-256, cmap readability, script coverage, duplicate detection, and JSON/CSV/Markdown outputs.
  • Inventory current runtime font corpus: outputs/font-inventory-current-2026-06-16 has 17 font files, 7 cmap-readable fonts, and 0 duplicate SHA groups.
  • Inventory /Users/cdjk/github/llm/ltk/fonts.zip: found janaki.ttf and madan.ttf; both are cmap-readable but do not match registry Unicode ranges, so they stay legacy/conversion leads rather than OCR rendering fonts.
  • Probed cdjk@100.110.148.43 under ~/github/llm for font assets and copied direct font files into ignored data/fonts/remote-cdjk-2026-06-16.
  • Inventory remote-copied fonts: outputs/font-inventory-remote-cdjk-2026-06-16 has 4 readable fonts. Only Noto Sans Devanagari covers registered Devanagari targets; the 8-bit Devanagari font stays a legacy conversion lead.
  • Download and inventory Google Fonts OFL batch: Kanchenjunga Regular/Bold for Kirat Rai, Noto Sans Tirhuta for Tirhuta, Noto Serif Devanagari for print-style Devanagari, and Noto Sans Ol Chiki Variable for Santhali style diversity.
  • Add configs/nepal-font-assets-v1.json plus ocrtech prepare-font-assets so the Google Fonts batch is reproducibly provisioned with SHA-256 checks instead of manual curl commands.
  • Add ocrtech audit-font-asset-readiness to require passing asset preparation, font inventory, and all-script renderability evidence before treating a runtime font corpus as synthesis-ready.
  • Connect rendered synthesis audits to font readiness evidence via --font-readiness-report; rendered rows now fail training-bound audits when their font_sha256 is not in the passed readiness manifest.
  • Reaudit the Namdhinggo-rendered Limbu/Sirijonga train/eval split against Namdhinggo font readiness evidence: train=150, eval=27, issues=0, warnings=0.
  • Add deterministic rendered-line degradation profiles for PaddleOCR recognizer training: clean, scan, low_light, uneven_light, glare_light, camera_left, camera_right, camera_top, camera_bottom, and phone_photo. Every generated image now records the selected profile, seed, and parameters in manifest metadata.
  • Add ocrtech audit-rendered-degradation-split to verify expected degradation profiles per text hash and reject train/eval text_sha256 overlap in augmented rendered bundles.
  • Generate and audit the full Namdhinggo augmented Limbu/Sirijonga rendered split: 900 train rows, 162 eval rows, six profiles per text hash, issues=0, warnings=0.
  • Materialize the Limbu/Sirijonga augmentation ablation spec: configs/limbu-sirijonga-augmentation-ablation-v1.json, with clean, lighting, and camera arms plus training recipe directories for each arm.
  • Audit all ablation arms for profile balance and train/eval text-hash isolation: clean 150/27, lighting 450/81, camera 600/108, all issues=0, warnings=0.
  • Include illumination and capture-angle robustness in the first Sirijonga augmentation plan: low_light, uneven_light, glare_light, camera_left, camera_right, camera_top, camera_bottom, and phone_photo are now deterministic profiles with recorded seeds and parameters.
  • Upgrade camera/phone-photo profiles from affine shear to deterministic perspective transforms, and require source_quad, output_quad, and perspective_coefficients in rendered-manifest audits.
  • Tighten rendered-manifest audits so lighting profiles must record numeric brightness/contrast, gradient/blend, or glare metadata instead of relying on profile names alone.
  • Add ocrtech audit-augmentation-ablation so Limbu/Sirijonga synthetic ablation arms cannot be treated as promotion-ready without a passed real Gorkhapatra verified-reference review audit.
  • Run the current ablation promotion audit: passed=False arms=3 real_evidence=0 verified_references=0 issues=1. All synthetic arms pass internally, but the intended blocker is missing real verified Gorkhapatra evidence.
  • Run a controlled Limbu/Sirijonga augmentation ablation: clean-only vs lighting-only vs camera-angle/phone-photo mix, evaluated on real Gorkhapatra/Naya Nepal pages and held-out synthetic rows without text-hash leakage.
  • Promote coverage-proven Google Fonts candidates into languages/nepal-v1.json and rerun all-script renderability: passed=True scripts=22 candidates=58 rendered=0.
  • Collect and inventory additional print-style Unicode Devanagari, Ol Chiki, Kirat Rai, Tibetan/Tamyig, and other Nepal-script fonts with license notes before using them for synthesis diversity.

Gorkhapatra Validation Path

  • Start from https://gorkhapatraonline.com/publications/naya-nepal.
  • Support paginated Naya Nepal publication discovery with audit-gorkhapatra-language-pages --publication-pages N; Limbu appears on page 9 in the 2026-06-16 live scan.
  • Capture article publication dates and language labels.
  • Query the epaper endpoint for the matching publication date: https://epaper.gorkhapatraonline.com/epaper-list?...&slug=gorkhapatra.
  • Download the direct PDF from the viewer file= path, not the PDF.js shell.
  • Search/render the language page, including bhasa pristha / भाषा पृष्ठ candidates.
  • Extract PDF-native text spans, font names, bboxes, codepoints, and embedded fonts before OCR/manual transcription. This is now implemented as ocrtech extract-gorkhapatra-language-page-pdf-text.
  • Run the PDF-native extractor on the current Limbu pack: passed=True samples=2/2 spans=776 target_spans=304 fonts=10 warnings=0. Page 6 (gorkhapatra-language-page-0001-01) has 304 NamdhinggoSILL spans and extracted embedded font UFNWLJ+NamdhinggoSILL; page 7 has 0 target-font spans.
  • Compare the blog image, PDF page, and PDF font metadata to choose the exact page candidate. Current conclusion: page 6 is the Limbu/Sirijonga page; page 7 is the same-date non-Limbu page.
  • Build or locate a Namdhinggo/Gorkhapatra legacy encoding converter to Unicode Limbu (U+1900..U+194F) so extracted PDF spans can become reference text instead of manual transcription burden.
  • Finalize only after manual review accepts exactly one candidate for the publication/language/date group.
  • Add a candidate-language filter to the Gorkhapatra language-page review audit/finalize commands so Limbu can be gated independently from other accepted Naya Nepal languages on the same PDF.
  • Run the filtered Limbu verified-reference audit: passed=False samples=2 groups=1 accepted=1 rejected=1 issues=1 warnings=0. The only blocker is empty verified reference text for gorkhapatra-language-page-0001-01.
  • Add a transcription work-order generator for accepted Gorkhapatra language-page rows so manual labeling is explicit, auditable, and scoped by target language.
  • Generate the Limbu work order: passed=True samples=1 blocked=1 verified=0 warnings=0. The selected work item is gorkhapatra-language-page-0001-01, page 6 of the 2083-02-12 Gorkhapatra PDF.
  • Correct the work-order priority: for Gorkhapatra epapers, PDF-native extraction is upstream of OCR and manual transcription. The transcription dashboard is fallback/verification only, especially for Limbu/Sirijonga where OCR candidates are currently invalid.
  • Add prepare-gorkhapatra-language-page-assisted-references for non-claim-ready machine OCR draft references. Drafts preserve reference_status=draft, claim_evidence_eligible=false, machine_generated=true, and requires_human_review=true.
  • Add prepare-gorkhapatra-language-page-ocr-sidecars so accepted Gorkhapatra rows can produce reproducible OCR sidecars before assisted reference drafting.
  • Run PaddleOCR sidecar generation for the current Limbu page on k2: pages=1, lines=299, nonzero_bbox=299, average_confidence=0.5347613170992171.
  • Rerun the assisted-reference command on the current Limbu page: passed=True samples=1 assisted=1 failed=0 skipped=3 warnings=0. The draft now contains 298 non-empty OCR text lines and remains non-claim-ready.
  • Add prepare-gorkhapatra-language-page-verification-bundle for line-level human correction sheets from assisted references.
  • Add apply-gorkhapatra-language-page-verification-bundle to convert a completed line-review CSV into verified .ref.json files and an updated page-review CSV, while blocking pending, needs_context, unknown, or empty reviewed rows.
  • Tighten verified-reference application so claim-eligible references also require explicit table, figure, and caption review declarations. Checked absent is acceptable; silently inheriting empty draft arrays is not.
  • Generate the current Limbu verification bundle: passed=True samples=1 lines=298 crops=298 missing_bboxes=0 warnings=0. The CSV review sheet and line crops are ready for human correction.
  • Add an interactive HTML review dashboard to every verification bundle so reviewers can scan line crops, ids, order, confidence, and OCR text, set review_status, enter corrected text, add notes, and export a reviewed CSV that still must pass the audit/apply gates.
  • Add audit-gorkhapatra-language-page-verification-csv for dry-run status counts and blocker reports before writing verified references.
  • Add split-gorkhapatra-language-page-verification-csv to produce reviewer-sized CSV/HTML batches while preserving line ids and order.
  • Add merge-gorkhapatra-language-page-verification-batches so reviewed batch CSVs can be recombined only when every original (sample_id, line_id) appears exactly once.
  • Add assign-gorkhapatra-language-page-verification-batches so split batches have an explicit reviewer/due-date ledger for paper-grade provenance.
  • Run the real Limbu split/merge smoke: split passed=True samples=1 lines=298 batches=6 batch_size=50 warnings=0; untouched-batch merge passed=True source_lines=298 merged_lines=298 batch_files=6 issues=0 warnings=0; merged audit still blocks with pending=298.
  • Run the current Limbu verification CSV audit: passed=False samples=1 lines=298 ready=0 dropped=0 blocked=298 issues=298 warnings=0; all rows remain pending.
  • Write structured references with text, reading order, tables, figures, captions, and Markdown where present.
  • Run audit-gorkhapatra-language-page-review and audit-references before using the pack in claim-grade validation.

Paper Evidence Checklist

  • Every benchmark report records input hashes, output artifact hashes, model-card hashes, latency, memory, and fallback usage.
  • Every public or real eval row has a parsed and audited reference.
  • Claim review rejects stale artifacts, missing references, malformed structured references, and synthetic-only evidence.
  • Error analysis includes failures by language, script, layout type, degradation, table presence, and fallback trigger reason.
  • Paper tables separate OCR metrics, document-structure metrics, resource metrics, and fallback metrics.

Limbu/Sirijonga Current Evidence

  • Built split-safe Limbu Unicode synthesis text: outputs/sirijonga-limbu-split-2026-06-16/train.jsonl and outputs/sirijonga-limbu-split-2026-06-16/eval.jsonl.
  • Split policy is split_synthesis_text:text_sha256:seed=17:eval_ratio=0.15, with 170 train rows, 30 eval rows, and 200 text-hash groups.
  • Rendered the pure Limbu/Sirijonga subset with font provenance: 150 train lines and 27 eval lines passed rendered-line audits using Noto Sans Limbu.
  • Rerendered the same split with Namdhinggo Regular; this is now the preferred Sirijonga training bundle.
  • Reaudited the Namdhinggo-rendered split with --font-readiness-report outputs/font-asset-readiness-namdhinggo-2026-06-16/font-asset-readiness.json; the training bundle is now tied to reproducible Namdhinggo asset provenance.
  • Regenerated the preferred Namdhinggo split with explicit degradation profiles after confirming the clean split remained reproducible. The degradation split audit confirms clean/degraded variants stay on the same split side.
  • Add a Limbu/Sirijonga PaddleOCR dictionary covering U+1900..U+194F plus accepted punctuation.
  • Audit that dictionary against the Namdhinggo-rendered train/eval label files: 76 dictionary characters, 68 required Limbu-range characters, 64 observed label characters, 0 missing required, 0 missing labels. Label spaces are covered by PaddleOCR use_space_char=true, not by a literal whitespace dictionary row.
  • Find or label real Gorkhapatra/Naya Nepal Sirijonga pages before any claim-grade evaluation.
  • Found the real Limbu Naya Nepal item: https://gorkhapatraonline.com/news/207555, date 2083-02-12, article image https://gorkhapatraonline.com/storage/media/360954/conversions/6-original.jpg.
  • Matched that date to Gorkhapatra epaper PDF 2026-05-26-03-43-57-2083-02-12.pdf, SHA-256 afe1d3d2e64248745d94d511e5ca456abddd3ad828480baff87263029b987cd8.
  • Rendered same-date page candidates and visually disambiguated: page 6 is Limbu/Sirijonga, page 7 is Majhi/Devanagari.
  • Created an accepted-row reviewer bundle for manual reference labeling: data/limbu-gorkhapatra-language-page-reviewer-bundle-2026-06-16, with 2 images and 2 draft references.
  • Created a Limbu transcription work order: data/limbu-gorkhapatra-language-page-transcription-work-order-2026-06-16. It confirms one accepted Limbu page is blocked only by the empty reference JSON, not by missing image/PDF provenance.
  • Created a Limbu assisted-reference draft run: data/limbu-gorkhapatra-language-page-assisted-references-2026-06-16. It now contains PaddleOCR-assisted text suggestions from k2 and keeps the draft non-claim-ready.
  • Created a Limbu line-level verification bundle: data/limbu-gorkhapatra-language-page-verification-bundle-2026-06-16, with 298 line crops, a pending review CSV, and an HTML review contact sheet.
  • Confirmed the correct PDF-native browser review surface: file:///Users/cdjk/github/llm/ocr-tech/data/limbu-gorkhapatra-language-page-pdf-text-2026-06-16/gorkhapatra-language-page-pdf-text-dashboard.html. Use this full-page clickable PDF-span dashboard for verification. Do not use the PDF-native verification-bundle HTML as the review UI; that was identified as the wrong/broken interface.
  • Fill the verification CSV with explicit accept, corrected, or drop line statuses, either directly or via split-gorkhapatra-language-page-verification-csv batches. If batches are used, merge them with merge-gorkhapatra-language-page-verification-batches. Then run audit-gorkhapatra-language-page-verification-csv, then run apply-gorkhapatra-language-page-verification-bundle to produce verified references and an updated review CSV with --tables-reviewed, --figures-reviewed, and --captions-reviewed after those structures are checked.
  • Fill verified text/layout references for the accepted Limbu page; strict review audit currently fails as intended until that reference is complete.
  • Re-run the Limbu-only gate after transcription: audit-gorkhapatra-language-page-review ... --candidate-language लिम्बू --require-verified-references.