ocr-projects-archive / OCR_INDRA /docs /INDRA_EXPERIMENTS.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
14.9 kB

Indra × Nepali — Experiment Plan (A–E)

Companion to docs/INDRA_INVESTIGATION.md. Everything runnable is designed around tools/indra_eval/eval_indra.py (works on CPU; the remote T4 simply makes batches fast). Remote-dependent sections are marked [B] and are blocked until SSH access to khalti-t4 (202.51.0.7) is restored; every command to run there is written out so the set can be executed as-is.

Fairness principle: recognizer comparisons use identical line crops (rec-lines mode); pipeline comparisons use page-level text. Both modes are built and smoke-tested (results/smoke/: CER 3.49% rec-only / 2.62% full on a synthetic page — indicational only).


Experiment A — zero-shot Indra vs deployed PP-OCRv5 (do FIRST) [B with A.1 runnable]

Goal: close "does the ~1.24 M-param Indra recognizer beat the deployed PP-OCRv5 Mobile Devanagari recognizer?" without training anything.

Materials (all remote): frozen eval sets ~/ocr/ocr-eval-20260924*/ + docs/DATA_LICENSING.md, the deployed paddle pipeline (~/ocr/ocr-deva-ft, tools/deva_rec_torch, tools/eval_metrics.py), the CLONE of this repo + models/indra-ocr-v3.1.1/ (copy indra/ and models/ up: ~46 MB — no dataset leaves the machine).

A.1 runnable from here already indep. of remote sets:

  • make_synth_page.py renders pages at matching font metrics for a sanity envelope (no migration of datasets needed: it was built here, scored here).

A.2 (remote) recognizer head-to-head on identical crops:

  1. Export the deployed pipeline's GT line boxes for an eval set once: PP-OCRv5 det results on the set → boxes jsonl ({"file", lines: [{i, text: GT, box}]}).
  2. Run our adapter on the same crops:
    PYTHONPATH=$HOME/ocr/OCR_INDRA/indra:$HOME/ocr/dep python3 \
      tools/indra_eval/eval_indra.py predict <page.jpg> \
      --boxes <gt_boxes.jsonl> --jsonl results/indra_rec/<set>.jsonl
    
    and score with the identical-crop baseline (PP-OCRv5 rec on the same cropped images saved as png), then cer both against GT.
  3. Report: micro/macro CER + WER + per-line detail + conf histogram;(any MIN_CONF-dropped lines are reported separately (they hit the full-pipeline number but not rec-lines).
  4. Also submit full-pipeline Indra (predict without --boxes) on the same set for the det+rec+order end-to-end number.

Pass/fail: Indra rec-lines CER < deployed rec CER on ≥ 2 of 3 eval sets, with per-set mean confidence reported. Also record det recall (lines found / GT lines) for the full-pipeline run.

Experiment B — fine-tune the recognizer (the supported path)

Goal: Adapt rec.onnx's weights to Nepali-specific distribution (conjunct-heavy printing, nukta, ₹, Devanagari digits in running text), retrain with model.safetensors as init, export an ONNX with the identical interface, drop-in replace.

Architecture requirements to reproduce byte-compatible behavior (from Investigation §1):

  • module: stem Conv(1→128, k32×4, s32×4) + BN + HardSwish → mask Mul → 10 × [dw 9-k1 g128 pad4 → expand 1×1 128→256 → HS → SE(masked-mean→256→64→256 sigmoid gate) → project 1×1 256→128 → +residual → mask Mul] → head 1×1 128→1678 → softmax → argmax.
  • Padding mask (arange(W/4) < lengths).float() with lengths = ceil(widths/4) (clip W/4); apply post-stem, at SE pools (mean over VALID steps only), and after each block's residual add. The mask is functional, not cosmetic: it keeps SE statistics from being diluted by padding.
  • Parameter mapping table: model.safetensors keys rec/<name> — stem.weight [128,1,32,4], stem.bias [128], stem_bn.{weight,bias, running_mean,running_var} [128], blocks.<i>.dw.weight [128,1,9] / dw.bias [128] (ONNX names onnx::Conv_{369,370};372,373,…396,397), blocks.<i>.expand.{weight,bias} [256,128], blocks.<i>.se.fc1/fc2, blocks.<i>.project.{weight,bias} [128,256], head.weight [1678,128,1], head.bias [1678].
  • Loss: CTC (log_softmax over [B,T,1678]; blank id 0; input_lengths = lengths, target = vocab.encode(canonical(text)) IDs list; NFC-normalize exactly as vocab.py:42).
  • Optimizer: AdamW, lr 3e-4 cosine, 10–20k steps, batch 32 lines by width-bucket, crop 32px — the pipeline above runs on 1× A100/40 GB in minutes per epoch for our line counts ([B] exact data volume from `/ocr/ocr-deva-ft/data`).

Unique risk (verified): the trainer's crop augmentation recipe is closed; starting too aggressively (rotate, tone) would shift the input distribution away from crop_line's exact ops. Mitigation: augmentations limited to elastic/scale jitter of ±5% consistent with PAD_Y/PAD_X bend, train/eval identical crop code (shared module imported from rec_preproc.py — the DOC that matters).

Export/eval: torch → ONNX with same outputs (ids,conf,lengths) or a logits-only export + a 15-line Python decode; either way validate against rec.onnx outputs on 512 in-crops before scoring (exactness test: max |Δconf| < 1e-3 on identical inputs).

Verified offline 2026-10-20 (torch 2.14.1+cpu / onnxruntime 1.31.0, tools/exp/verify_bundle.sh): model.safetensors → IndraRec (1,236,366 tensors) → ONNX export matches the shipped rec.onnx with 0 id mismatches, 0 lengths mismatches, max |Δconf| 7.1e-6 on the 1,590-step probe battery; the post-train gate (torch .pt vs its own export) passes at the same tolerance; run_rec.py reproduces the recorded smoke CER (3.49 %) bit-for-bit. The exporter forces the legacy TorchScript path (dynamo=False) because torch ≥ 2.6 defaults to the torch.export exporter, which needs the onnxscript package.

Acceptance: fine-tuned rec-lines CER < zero-shot rec-lines CER on all 3 frozen sets by ≥ 2 points relative, on Devanagari digit-heavy and conjunct-heavy slices specifically (those are where the smoke run showed weakness: सङ्केत→सड्लेत, ₹→र, ऋतु→तऋहतु).

Experiment C — fine-tune the line detector / order model

Verdict from Investigation §3: blocked by shipped-artifact reality — det.onnx/order.onnx come with NO state dict and NO recipe. Two paths:

  1. Torch re-implementation from scratch using the target specs that ARE in the repo (det_post.encode_targets → line core logit + top/bottom distance regression; order_post.label_order → 10 block classes + pairwise salience). Cost: reimplementing the backbone (which is unknown — the ONNX graphs would need the same node-by-node reverse archaeology as rec.onnx: feasible, we did it, but heavy) + a data pipeline + synth label generator. Not recommended as the first move — zero-shot + rec-only fine-tuning captures most value: the det model already detects 10/10 synthetic lines and machine-printed Devanagari well.
  2. Skip det/order entirely; if detection quality on real scan material turns out to be the bottleneck in A-full, locally replace with our known corresponding PP-OCR det model and keep only the Indra rec — that is Experiment D.

Experiment D — hybrid: keep PP-OCR det, swap rec → Indra

Goal: combine a known-good Devanagari line finder (PP-OCRv5 det) with the Indra recognizer under identical crops, and vice-versa. Outputs line boxes jsonl exactly like A.2-step 1. This is a 1-day experiment with zero training and makes the fair-comparison claim airtight: if PP-det + Indra-rec beats PP-det + PP-rec on the same crops, the recognizer itself is the win; if not, Indra's advantage (if any) lives elsewhere (orientation/order/layout).

Experiment E — productionization / distill for WSL CPU box

Goal: a deployable artifact for the 8-core/3 GB (or T4) runtime:

  • Indra package supports rec.int8.onnx automatically (pipeline.py:195–200) — quantize the rec model once (no retraining), then re-run Experiment A with the int8 model: acceptance: CER regression ≤ +0.3 pts absolute and ≥ 2× throughput.
  • Measured local guidance (2026-10-20, WSL CPU, threads=4, smoke crops; indicative, not a benchmark): ORT dynamic (weights-only) quantization is a trap for this Conv+SE model — 0.69–0.27× fp32 throughput (ConvInteger + per-activation quant/dequant dominates) though greedy ids stay identical and max |Δconf| = 0.077. Static QDQ uint8 (per-tensor, MinMax, 64 calib crops) built 3.5× smaller (4.98 → 1.43 MB) and ran ~1.4× faster than fp32 — on a stronger box the gap should widen; the ≥2× acceptance must be re-measured there. Tool: tools/exp/e_int8_quantize.py (static, calibration through the shared crop_line; --check-smoke reports id/conf drift; writes a JSON report). Use real line crops for calibration (the local test used noise → id drift 56 steps on the smoke page; real calibration should cut that).
  • Latency budget on our box is meaningless (WSL starved) — measure only on the T4: lines/s at long-side 1200/1600/2048 (det scales with page size).
  • Memory: with enable_cpu_mem_arena=False (already set in the package), line-batched rec is flat per page.

§11 Data plan for fine-tuning (Experiment B)

  • Source: the same frozen train corpus that produced the deployed fine-tuned PP-OCR model (~/ocr/ocr-deva-ft/data — [B] inventory is on remote; docs/DEVANAGARI_RESOURCES_INVENTORY.md documents it). No dataset is copied off the remote machine — training artifacts/results files transfer, not eval sets.
  • Emphasis slices for Nepali: conjunct-heavy book/print pages, Devanagari digit blocks, ₹-annotated amounts, मध्य-मस useful — determined by the per-line CER profile of A-zero (error concentration, not vibes).
  • Same-crop exactness: train the augmented pages through the exact crop_line/pad_batch code (import from the cloned package), so zero-shot and fine-tuned are measured through one code path.
  • Leakage (§12): any train page rendered by the deployed pipeline must be excluded from ALL three frozen eval sets; confirm set boundaries by re-running tools/eval_metrics.py's pairing logic [B].

§12 Leakage review [B]

  1. Set purity: frozen eval sets must not overlap the rec fine-tune corpus. Verify by page-hash (sha256 of rendered page images), not filenames ([B] run on remote).
  2. Synth contamination: if synthetic rendered pages were part of ocr-deva-ft training and the eval sets contain natural scans only, no leakage; confirm the eval-set provenance doc. Rendered-by-us (make_synth_page.py) pages are NEVER used for training or reporting.
  3. Adaptation leakage: Experiment D swaps components inside a pipeline built on eval sets — acceptable for comparison because GT boxes come from the DETECTOR of the deployed pipeline on EVAL images only, and neither component is trained on them.
  4. Font leakage: if fonts in the training corpus are also in the eval sets' print source (same family), CER estimates are optimistic for BOTH systems equally — acceptable for A/B ranking, not for absolute claims; state it in the report.

§13 Licensing matrix [B for dataset rows]

Item License Fine-tune allowed? Deploy allowed? Notes
Indra weights v3.1.1 AI PUBS Open RAIL-M (modified) YES, explicit (§II ¶36) YES (restrictions = Attachment A misuse list; internal OCR unaffected) Derivative weights → same license + notice on redistribution
indra_ocr code Apache-2.0 n/a YES
PaddleOCR (baseline) Apache-2.0 YES (we already fine-tune it) YES current deployed
PP-OCRv5 Mobile Devanagari weights Apache-2.0 YES YES deployed
Nepali train corpus per docs/DATA_LICENSING.md [B] pending review — no removal of datasets from remote
Frozen eval sets per docs/DATA_LICENSING.md [B] evaluation only —
Noto Sans Devanagari (testrender only) OFL 1.1 YES (fonts embedding fine) test asset only tools/indra_eval/testdata/

§14 Deployment comparison

Axis Indra v3.1.1 (rec-only swap / full) PP-OCRv5 Mobile Devanagari deployed
Recognizer params 1.236 M (verified) rec ~similar class (Mobile range; exact count from remote bench)
Runtime onnxruntime CPU (already installed here) paddle (remote)
Extra machinery needed none — single ONNX + meta.json vocab JSON paddle + FineTuned assets + converter glue
Latency (lines/s, T4 CPU) measure (A/D/E) measure
Deterministic CPU memory flat batch (mask) paddle per-page buffers
Order/reading model included (order.onnx; optional) not used for plain OCR flow
Honest staffing: fine-tune effort reimplement trainer (~1–2 days) already exists for Paddle
Dependency risk pinned HF revision + local files paddle version drift

§17 Decision tree

  1. Run A (zero-shot rec-lines + full) on the frozen sets.
    • If Indra rec CER < deployed rec CER on 2 of 3 sets: proceed to the hybrid experiment D (PP-det + Indra-rec); if D holds the win, plan a rec-swap deployment trial alongside the existing PP pipeline(1 week).
    • If Indra rec CER ≥ deployed rec CER: record the gap per slice. If the gap is train-material (conjuncts/₹/digits): run B (fine-tune rec from safetensors, ~10–20k steps); expected to close a distribution-gap-shaped loss; decide by ΔCER on the eval sets.
  2. Regardless of A's outcome, the graceful fallback stands: bringing Indra recognizer into our PP-OCR det pipeline is a drop-in component (same crops→ink-map contract), so nothing about A/B/D commits us.
  3. Det/order fine-tuning (C) stays out of scope until det-on-real-scan failure is demonstrated in A-full (det recall < line recall of deployed).
  4. Never touch: datasets on the remote, the deployed pipeline's recipes, its eval scripts, or ~/.ssh/~/.bashrc. All experiments run in a separate directory (~/ocr/OCR_INDRA/), copying only code/weights in.

§18 Status summary

  • [V] runnable here end-to-end: smoke chain (synth page → rec-only → full → CER), all inputs/outputs/paths in Investigation §8.
  • [B] everything requiring khalti-t4: A.2 (frozen sets), B data, C skip decision input, licensing rows, E throughput on deployment hardware.
  • [V] fine-tune toolkit verified offline 2026-10-20 (tools/exp/verify_bundle.sh: export exactness pass; mechanics training + post-train gate pass; A/B orchestrations complete on a local stand-in set); int8 static/dynamic behaviour probed (dynamic = slower trap, static = ~1.4× faster, 3.5× smaller).
  • Open: WP det recall numbers, paddle latency comparison, int8 CER on real calibration + deployment-box throughput.