# Indra × Nepali — Experiment Plan (A–E) Companion to `docs/INDRA_INVESTIGATION.md`. Everything runnable is designed around `tools/indra_eval/eval_indra.py` (works on CPU; the remote T4 simply makes batches fast). Remote-dependent sections are marked **[B]** and are blocked until SSH access to `khalti-t4` (202.51.0.7) is restored; every command to run there is written out so the set can be executed as-is. Fairness principle: **recognizer comparisons use identical line crops** (rec-lines mode); **pipeline comparisons use page-level text**. Both modes are built and smoke-tested (`results/smoke/`: CER 3.49% rec-only / 2.62% full on a synthetic page — indicational only). --- ## Experiment A — zero-shot Indra vs deployed PP-OCRv5 (do FIRST) [B with A.1 runnable] **Goal:** close "does the ~1.24 M-param Indra recognizer beat the deployed PP-OCRv5 Mobile Devanagari recognizer?" without training anything. **Materials (all remote):** frozen eval sets `~/ocr/ocr-eval-20260924*/` + `docs/DATA_LICENSING.md`, the deployed paddle pipeline (`~/ocr/ocr-deva-ft`, `tools/deva_rec_torch`, `tools/eval_metrics.py`), the CLONE of this repo + `models/indra-ocr-v3.1.1/` (copy `indra/` and `models/` up: ~46 MB — no dataset leaves the machine). **A.1 runnable from here already indep. of remote sets:** - `make_synth_page.py` renders pages at **matching font metrics** for a sanity envelope (no migration of datasets needed: it was built here, scored here). **A.2 (remote) recognizer head-to-head on identical crops:** 1. Export the deployed pipeline's GT line boxes for an eval set once: PP-OCRv5 det results on the set → boxes jsonl (`{"file", lines: [{i, text: GT, box}]}`). 2. Run our adapter on the same crops: ``` PYTHONPATH=$HOME/ocr/OCR_INDRA/indra:$HOME/ocr/dep python3 \ tools/indra_eval/eval_indra.py predict \ --boxes --jsonl results/indra_rec/.jsonl ``` and score with the identical-crop baseline (PP-OCRv5 rec on the same cropped images saved as png), then `cer` both against GT. 3. Report: micro/macro CER + WER + per-line detail + conf histogram;(any `MIN_CONF`-dropped lines are reported separately (they hit the full-pipeline number but not rec-lines). 4. Also submit **full-pipeline** Indra (`predict` without `--boxes`) on the same set for the det+rec+order end-to-end number. **Pass/fail:** Indra rec-lines CER < deployed rec CER on ≥ 2 of 3 eval sets, with per-set mean confidence reported. Also record det recall (lines found / GT lines) for the full-pipeline run. ## Experiment B — fine-tune the recognizer (the supported path) **Goal:** Adapt `rec.onnx`'s weights to Nepali-specific distribution (conjunct-heavy printing, nukta, ₹, Devanagari digits in running text), retrain with `model.safetensors` as init, export an ONNX with the *identical* interface, drop-in replace. **Architecture requirements to reproduce byte-compatible behavior (from Investigation §1):** - module: stem Conv(1→128, k32×4, s32×4) + BN + HardSwish → mask Mul → 10 × [dw 9-k1 g128 pad4 → expand 1×1 128→256 → HS → SE(masked-mean→256→64→256 sigmoid gate) → project 1×1 256→128 → +residual → mask Mul] → head 1×1 128→1678 → softmax → argmax. - Padding mask `(arange(W/4) < lengths).float()` with `lengths = ceil(widths/4)` (clip W/4); apply post-stem, at SE pools (mean over VALID steps only), and after each block's residual add. **The mask is functional, not cosmetic: it keeps SE statistics from being diluted by padding.** - Parameter mapping table: `model.safetensors` keys `rec/` — `stem.weight` [128,1,32,4], `stem.bias` [128], `stem_bn.{weight,bias, running_mean,running_var}` [128], `blocks..dw.weight` [128,1,9] / `dw.bias` [128] (ONNX names `onnx::Conv_{369,370};372,373,…396,397`), `blocks..expand.{weight,bias}` [256,128], `blocks..se.fc1/fc2`, `blocks..project.{weight,bias}` [128,256], `head.weight` [1678,128,1], `head.bias` [1678]. - Loss: CTC (log_softmax over [B,T,1678]; blank id 0; input_lengths = lengths, target = `vocab.encode(canonical(text))` IDs list; NFC-normalize exactly as `vocab.py:42`). - Optimizer: AdamW, lr 3e-4 cosine, ~10–20k steps, batch 32 lines by width-bucket, crop 32px — the pipeline above runs on 1× A100/40 GB in minutes per epoch for our line counts ([B] exact data volume from `~/ocr/ocr-deva-ft/data`). **Unique risk (verified):** the trainer's crop augmentation recipe is closed; starting too aggressively (`rotate`, `tone`) would shift the input distribution away from `crop_line`'s exact ops. Mitigation: augmentations limited to elastic/scale jitter of ±5% consistent with `PAD_Y/PAD_X` bend, train/eval identical crop code (shared module imported from `rec_preproc.py` — the DOC that matters). **Export/eval:** torch → ONNX with same outputs (`ids,conf,lengths`) or a logits-only export + a 15-line Python decode; either way validate against `rec.onnx` outputs on 512 in-crops before scoring (**exactness test**: max |Δconf| < 1e-3 on identical inputs). **Verified offline 2026-10-20** (torch 2.14.1+cpu / onnxruntime 1.31.0, `tools/exp/verify_bundle.sh`): `model.safetensors` → IndraRec (1,236,366 tensors) → ONNX export matches the shipped `rec.onnx` with **0 id mismatches, 0 lengths mismatches, max |Δconf| 7.1e-6** on the 1,590-step probe battery; the post-train gate (torch .pt vs its own export) passes at the same tolerance; `run_rec.py` reproduces the recorded smoke CER (3.49 %) bit-for-bit. The exporter forces the legacy TorchScript path (`dynamo=False`) because torch ≥ 2.6 defaults to the torch.export exporter, which needs the `onnxscript` package. **Acceptance:** fine-tuned rec-lines CER < zero-shot rec-lines CER on all 3 frozen sets by ≥ 2 points relative, on Devanagari digit-heavy and conjunct-heavy slices specifically (those are where the smoke run showed weakness: `सङ्केत→सड्लेत`, `₹→र`, `ऋतु→तऋहतु`). ## Experiment C — fine-tune the line detector / order model **Verdict from Investigation §3: blocked by shipped-artifact reality** — `det.onnx`/`order.onnx` come with NO state dict and NO recipe. Two paths: 1. **Torch re-implementation from scratch** using the target specs that ARE in the repo (`det_post.encode_targets` → line core logit + top/bottom distance regression; `order_post.label_order` → 10 block classes + pairwise salience). Cost: reimplementing the backbone (which is unknown — the ONNX graphs would need the same node-by-node reverse archaeology as rec.onnx: feasible, we did it, but heavy) + a data pipeline + synth label generator. **Not recommended as the first move** — zero-shot + rec-only fine-tuning captures most value: the det model already detects 10/10 synthetic lines and machine-printed Devanagari well. 2. **Skip det/order entirely**; if detection quality on real scan material turns out to be the bottleneck in A-full, locally replace with our known corresponding PP-OCR det model and keep only the Indra rec — that is Experiment D. ## Experiment D — hybrid: keep PP-OCR det, swap rec → Indra **Goal:** combine a known-good Devanagari line finder (PP-OCRv5 det) with the Indra recognizer under identical crops, and vice-versa. Outputs line boxes jsonl exactly like A.2-step 1. This is a 1-day experiment **with zero training** and makes the fair-comparison claim airtight: if `PP-det + Indra-rec` beats `PP-det + PP-rec` on the same crops, the recognizer itself is the win; if not, Indra's advantage (if any) lives elsewhere (orientation/order/layout). ## Experiment E — productionization / distill for WSL CPU box **Goal:** a deployable artifact for the 8-core/3 GB (or T4) runtime: - Indra package supports `rec.int8.onnx` automatically (`pipeline.py:195–200`) — quantize the rec model once (no retraining), then re-run Experiment A with the int8 model: acceptance: CER regression ≤ +0.3 pts absolute and ≥ 2× throughput. - **Measured local guidance (2026-10-20, WSL CPU, threads=4, smoke crops; indicative, not a benchmark):** ORT *dynamic* (weights-only) quantization is a **trap** for this Conv+SE model — 0.69–0.27× fp32 throughput (ConvInteger + per-activation quant/dequant dominates) though greedy ids stay identical and max |Δconf| = 0.077. *Static* QDQ uint8 (per-tensor, MinMax, 64 calib crops) built 3.5× smaller (4.98 → 1.43 MB) and ran **~1.4× faster** than fp32 — on a stronger box the gap should widen; the ≥2× acceptance must be re-measured there. Tool: `tools/exp/e_int8_quantize.py` (static, calibration through the shared `crop_line`; `--check-smoke` reports id/conf drift; writes a JSON report). Use real line crops for calibration (the local test used noise → id drift 56 steps on the smoke page; real calibration should cut that). - Latency budget on our box is meaningless (WSL starved) — measure only on the T4: lines/s at long-side 1200/1600/2048 (det scales with page size). - Memory: with `enable_cpu_mem_arena=False` (already set in the package), line-batched rec is flat per page. --- ## §11 Data plan for fine-tuning (Experiment B) - Source: the same frozen train corpus that produced the deployed fine-tuned PP-OCR model (`~/ocr/ocr-deva-ft/data` — [B] inventory is on remote; `docs/DEVANAGARI_RESOURCES_INVENTORY.md` documents it). **No dataset is copied off the remote machine** — training artifacts/results files transfer, not eval sets. - Emphasis slices for Nepali: conjunct-heavy book/print pages, Devanagari digit blocks, ₹-annotated amounts, मध्य-मस useful — determined by the per-line CER profile of A-zero (error concentration, not vibes). - Same-crop exactness: train the augmented pages through the exact `crop_line`/`pad_batch` code (import from the cloned package), so zero-shot and fine-tuned are measured through one code path. - Leakage (§12): any *train* page rendered by the deployed pipeline must be excluded from ALL three frozen eval sets; confirm set boundaries by re-running `tools/eval_metrics.py`'s pairing logic [B]. ## §12 Leakage review [B] 1. **Set purity:** frozen eval sets must not overlap the rec fine-tune corpus. Verify by page-hash (sha256 of rendered page images), not filenames ([B] run on remote). 2. **Synth contamination:** if synthetic rendered pages were part of `ocr-deva-ft` training and the eval sets contain natural scans only, no leakage; confirm the eval-set provenance doc. Rendered-by-us (`make_synth_page.py`) pages are NEVER used for training or reporting. 3. **Adaptation leakage:** Experiment D swaps components inside a pipeline built on eval sets — acceptable for comparison because GT boxes come from the DETECTOR of the deployed pipeline on EVAL images only, and neither component is trained on them. 4. **Font leakage:** if fonts in the training corpus are also in the eval sets' print source (same family), CER estimates are optimistic for BOTH systems equally — acceptable for A/B ranking, not for absolute claims; state it in the report. ## §13 Licensing matrix [B for dataset rows] | Item | License | Fine-tune allowed? | Deploy allowed? | Notes | |---|---|---|---|---| | Indra weights v3.1.1 | AI PUBS Open RAIL-M (modified) | **YES, explicit** (§II ¶36) | YES (restrictions = Attachment A misuse list; internal OCR unaffected) | Derivative weights → same license + notice on redistribution | | `indra_ocr` code | Apache-2.0 | n/a | YES | | | PaddleOCR (baseline) | Apache-2.0 | YES (we already fine-tune it) | YES | current deployed | | PP-OCRv5 Mobile Devanagari weights | Apache-2.0 | YES | YES | deployed | | Nepali train corpus | per `docs/DATA_LICENSING.md` [B] | pending review | — | no removal of datasets from remote | | Frozen eval sets | per `docs/DATA_LICENSING.md` [B] | evaluation only | — | | | Noto Sans Devanagari (testrender only) | OFL 1.1 | YES (fonts embedding fine) | test asset only | `tools/indra_eval/testdata/` | ## §14 Deployment comparison | Axis | Indra v3.1.1 (rec-only swap / full) | PP-OCRv5 Mobile Devanagari deployed | |---|---|---| | Recognizer params | **1.236 M** (verified) | rec ~similar class (Mobile range; exact count from remote bench) | | Runtime | onnxruntime CPU (already installed here) | paddle (remote) | | Extra machinery needed | **none** — single ONNX + meta.json vocab JSON | paddle + FineTuned assets + converter glue | | Latency (lines/s, T4 CPU) | measure (A/D/E) | measure | | Deterministic CPU memory | flat batch (mask) | paddle per-page buffers | | Order/reading model | included (order.onnx; optional) | not used for plain OCR flow | | Honest staffing: fine-tune effort | reimplement trainer (~1–2 days) | already exists for Paddle | | Dependency risk | pinned HF revision + local files | paddle version drift | ## §17 Decision tree 1. **Run A (zero-shot rec-lines + full) on the frozen sets.** - If Indra rec CER < deployed rec CER **on 2 of 3 sets**: proceed to the hybrid experiment D (PP-det + Indra-rec); if D holds the win, plan a **rec-swap deployment trial** alongside the existing PP pipeline(1 week). - If Indra rec CER ≥ deployed rec CER: record the gap per slice. If the gap is train-material (conjuncts/₹/digits): run **B** (fine-tune rec from safetensors, ~10–20k steps); expected to close a distribution-gap-shaped loss; decide by ΔCER on the eval sets. 2. **Regardless of A's outcome,** the graceful fallback stands: bringing Indra recognizer into our PP-OCR det pipeline is a **drop-in component** (same crops→ink-map contract), so nothing about A/B/D commits us. 3. **Det/order fine-tuning (C) stays out of scope** until det-on-real-scan failure is demonstrated in A-full (`det recall < line recall of deployed`). 4. **Never** touch: datasets on the remote, the deployed pipeline's recipes, its eval scripts, or `~/.ssh`/`~/.bashrc`. All experiments run in a separate directory (`~/ocr/OCR_INDRA/`), copying only code/weights in. ## §18 Status summary - [V] runnable here end-to-end: smoke chain (synth page → rec-only → full → CER), all inputs/outputs/paths in Investigation §8. - [B] everything requiring `khalti-t4`: A.2 (frozen sets), B data, C skip decision input, licensing rows, E throughput on deployment hardware. - [V] fine-tune toolkit verified offline 2026-10-20 (`tools/exp/verify_bundle.sh`: export exactness pass; mechanics training + post-train gate pass; A/B orchestrations complete on a local stand-in set); int8 static/dynamic behaviour probed (dynamic = slower trap, static = ~1.4× faster, 3.5× smaller). - Open: WP det recall numbers, paddle latency comparison, int8 CER on real calibration + deployment-box throughput.