Download OCR_INDRA/docs/INDRA_EXPERIMENTS.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 14.9 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/OCR_INDRA/docs/INDRA_EXPERIMENTS.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/OCR_INDRA/docs/INDRA_EXPERIMENTS.md
-
curl -L -o INDRA_EXPERIMENTS.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/OCR_INDRA/docs/INDRA_EXPERIMENTS.md
Indra × Nepali — Experiment Plan (A–E)
Companion to docs/INDRA_INVESTIGATION.md. Everything runnable is designed
around tools/indra_eval/eval_indra.py (works on CPU; the remote T4 simply
makes batches fast). Remote-dependent sections are marked [B] and are
blocked until SSH access to khalti-t4 (202.51.0.7) is restored; every
command to run there is written out so the set can be executed as-is.
Fairness principle: recognizer comparisons use identical line crops
(rec-lines mode); pipeline comparisons use page-level text. Both modes are
built and smoke-tested (results/smoke/: CER 3.49% rec-only / 2.62% full on a
synthetic page — indicational only).
Experiment A — zero-shot Indra vs deployed PP-OCRv5 (do FIRST) [B with A.1 runnable]
Goal: close "does the ~1.24 M-param Indra recognizer beat the deployed PP-OCRv5 Mobile Devanagari recognizer?" without training anything.
Materials (all remote): frozen eval sets ~/ocr/ocr-eval-20260924*/ +
docs/DATA_LICENSING.md, the deployed paddle pipeline
(~/ocr/ocr-deva-ft, tools/deva_rec_torch, tools/eval_metrics.py), the
CLONE of this repo + models/indra-ocr-v3.1.1/ (copy indra/ and models/
up: ~46 MB — no dataset leaves the machine).
A.1 runnable from here already indep. of remote sets:
make_synth_page.pyrenders pages at matching font metrics for a sanity envelope (no migration of datasets needed: it was built here, scored here).
A.2 (remote) recognizer head-to-head on identical crops:
- Export the deployed pipeline's GT line boxes for an eval set once:
PP-OCRv5 det results on the set → boxes jsonl
(
{"file", lines: [{i, text: GT, box}]}). - Run our adapter on the same crops:
and score with the identical-crop baseline (PP-OCRv5 rec on the same cropped images saved as png), thenPYTHONPATH=$HOME/ocr/OCR_INDRA/indra:$HOME/ocr/dep python3 \ tools/indra_eval/eval_indra.py predict <page.jpg> \ --boxes <gt_boxes.jsonl> --jsonl results/indra_rec/<set>.jsonlcerboth against GT. - Report: micro/macro CER + WER + per-line detail + conf histogram;(any
MIN_CONF-dropped lines are reported separately (they hit the full-pipeline number but not rec-lines). - Also submit full-pipeline Indra (
predictwithout--boxes) on the same set for the det+rec+order end-to-end number.
Pass/fail: Indra rec-lines CER < deployed rec CER on ≥ 2 of 3 eval sets, with per-set mean confidence reported. Also record det recall (lines found / GT lines) for the full-pipeline run.
Experiment B — fine-tune the recognizer (the supported path)
Goal: Adapt rec.onnx's weights to Nepali-specific distribution
(conjunct-heavy printing, nukta, ₹, Devanagari digits in running text), retrain
with model.safetensors as init, export an ONNX with the identical
interface, drop-in replace.
Architecture requirements to reproduce byte-compatible behavior (from Investigation §1):
- module: stem Conv(1→128, k32×4, s32×4) + BN + HardSwish → mask Mul → 10 × [dw 9-k1 g128 pad4 → expand 1×1 128→256 → HS → SE(masked-mean→256→64→256 sigmoid gate) → project 1×1 256→128 → +residual → mask Mul] → head 1×1 128→1678 → softmax → argmax.
- Padding mask
(arange(W/4) < lengths).float()withlengths = ceil(widths/4)(clip W/4); apply post-stem, at SE pools (mean over VALID steps only), and after each block's residual add. The mask is functional, not cosmetic: it keeps SE statistics from being diluted by padding. - Parameter mapping table:
model.safetensorskeysrec/<name>—stem.weight[128,1,32,4],stem.bias[128],stem_bn.{weight,bias, running_mean,running_var}[128],blocks.<i>.dw.weight[128,1,9] /dw.bias[128] (ONNX namesonnx::Conv_{369,370};372,373,…396,397),blocks.<i>.expand.{weight,bias}[256,128],blocks.<i>.se.fc1/fc2,blocks.<i>.project.{weight,bias}[128,256],head.weight[1678,128,1],head.bias[1678]. - Loss: CTC (log_softmax over [B,T,1678]; blank id 0; input_lengths = lengths,
target =
vocab.encode(canonical(text))IDs list; NFC-normalize exactly asvocab.py:42). - Optimizer: AdamW, lr 3e-4 cosine,
10–20k steps, batch 32 lines by width-bucket, crop 32px — the pipeline above runs on 1× A100/40 GB in minutes per epoch for our line counts ([B] exact data volume from `/ocr/ocr-deva-ft/data`).
Unique risk (verified): the trainer's crop augmentation recipe is closed;
starting too aggressively (rotate, tone) would shift the input
distribution away from crop_line's exact ops. Mitigation: augmentations
limited to elastic/scale jitter of ±5% consistent with PAD_Y/PAD_X bend,
train/eval identical crop code (shared module imported from rec_preproc.py
— the DOC that matters).
Export/eval: torch → ONNX with same outputs (ids,conf,lengths) or a
logits-only export + a 15-line Python decode; either way validate against
rec.onnx outputs on 512 in-crops before scoring (exactness test:
max |Δconf| < 1e-3 on identical inputs).
Verified offline 2026-10-20 (torch 2.14.1+cpu / onnxruntime 1.31.0,
tools/exp/verify_bundle.sh): model.safetensors → IndraRec (1,236,366
tensors) → ONNX export matches the shipped rec.onnx with 0 id mismatches,
0 lengths mismatches, max |Δconf| 7.1e-6 on the 1,590-step probe battery;
the post-train gate (torch .pt vs its own export) passes at the same tolerance;
run_rec.py reproduces the recorded smoke CER (3.49 %) bit-for-bit. The
exporter forces the legacy TorchScript path (dynamo=False) because torch ≥ 2.6
defaults to the torch.export exporter, which needs the onnxscript package.
Acceptance: fine-tuned rec-lines CER < zero-shot rec-lines CER on all
3 frozen sets by ≥ 2 points relative, on Devanagari digit-heavy and
conjunct-heavy slices specifically (those are where the smoke run showed
weakness: सङ्केत→सड्लेत, ₹→र, ऋतु→तऋहतु).
Experiment C — fine-tune the line detector / order model
Verdict from Investigation §3: blocked by shipped-artifact reality —
det.onnx/order.onnx come with NO state dict and NO recipe. Two paths:
- Torch re-implementation from scratch using the target specs that ARE
in the repo (
det_post.encode_targets→ line core logit + top/bottom distance regression;order_post.label_order→ 10 block classes + pairwise salience). Cost: reimplementing the backbone (which is unknown — the ONNX graphs would need the same node-by-node reverse archaeology as rec.onnx: feasible, we did it, but heavy) + a data pipeline + synth label generator. Not recommended as the first move — zero-shot + rec-only fine-tuning captures most value: the det model already detects 10/10 synthetic lines and machine-printed Devanagari well. - Skip det/order entirely; if detection quality on real scan material turns out to be the bottleneck in A-full, locally replace with our known corresponding PP-OCR det model and keep only the Indra rec — that is Experiment D.
Experiment D — hybrid: keep PP-OCR det, swap rec → Indra
Goal: combine a known-good Devanagari line finder (PP-OCRv5 det) with the
Indra recognizer under identical crops, and vice-versa. Outputs line boxes
jsonl exactly like A.2-step 1. This is a 1-day experiment with zero training
and makes the fair-comparison claim airtight: if PP-det + Indra-rec beats
PP-det + PP-rec on the same crops, the recognizer itself is the win; if not,
Indra's advantage (if any) lives elsewhere (orientation/order/layout).
Experiment E — productionization / distill for WSL CPU box
Goal: a deployable artifact for the 8-core/3 GB (or T4) runtime:
- Indra package supports
rec.int8.onnxautomatically (pipeline.py:195–200) — quantize the rec model once (no retraining), then re-run Experiment A with the int8 model: acceptance: CER regression ≤ +0.3 pts absolute and ≥ 2× throughput. - Measured local guidance (2026-10-20, WSL CPU, threads=4, smoke crops;
indicative, not a benchmark): ORT dynamic (weights-only) quantization is
a trap for this Conv+SE model — 0.69–0.27× fp32 throughput (ConvInteger +
per-activation quant/dequant dominates) though greedy ids stay identical and
max |Δconf| = 0.077. Static QDQ uint8 (per-tensor, MinMax, 64 calib crops)
built 3.5× smaller (4.98 → 1.43 MB) and ran ~1.4× faster than fp32 — on a
stronger box the gap should widen; the ≥2× acceptance must be re-measured
there. Tool:
tools/exp/e_int8_quantize.py(static, calibration through the sharedcrop_line;--check-smokereports id/conf drift; writes a JSON report). Use real line crops for calibration (the local test used noise → id drift 56 steps on the smoke page; real calibration should cut that). - Latency budget on our box is meaningless (WSL starved) — measure only on the T4: lines/s at long-side 1200/1600/2048 (det scales with page size).
- Memory: with
enable_cpu_mem_arena=False(already set in the package), line-batched rec is flat per page.
§11 Data plan for fine-tuning (Experiment B)
- Source: the same frozen train corpus that produced the deployed fine-tuned
PP-OCR model (
~/ocr/ocr-deva-ft/data— [B] inventory is on remote;docs/DEVANAGARI_RESOURCES_INVENTORY.mddocuments it). No dataset is copied off the remote machine — training artifacts/results files transfer, not eval sets. - Emphasis slices for Nepali: conjunct-heavy book/print pages, Devanagari digit blocks, ₹-annotated amounts, मध्य-मस useful — determined by the per-line CER profile of A-zero (error concentration, not vibes).
- Same-crop exactness: train the augmented pages through the exact
crop_line/pad_batchcode (import from the cloned package), so zero-shot and fine-tuned are measured through one code path. - Leakage (§12): any train page rendered by the deployed pipeline must be
excluded from ALL three frozen eval sets; confirm set boundaries by
re-running
tools/eval_metrics.py's pairing logic [B].
§12 Leakage review [B]
- Set purity: frozen eval sets must not overlap the rec fine-tune corpus. Verify by page-hash (sha256 of rendered page images), not filenames ([B] run on remote).
- Synth contamination: if synthetic rendered pages were part of
ocr-deva-fttraining and the eval sets contain natural scans only, no leakage; confirm the eval-set provenance doc. Rendered-by-us (make_synth_page.py) pages are NEVER used for training or reporting. - Adaptation leakage: Experiment D swaps components inside a pipeline built on eval sets — acceptable for comparison because GT boxes come from the DETECTOR of the deployed pipeline on EVAL images only, and neither component is trained on them.
- Font leakage: if fonts in the training corpus are also in the eval sets' print source (same family), CER estimates are optimistic for BOTH systems equally — acceptable for A/B ranking, not for absolute claims; state it in the report.
§13 Licensing matrix [B for dataset rows]
| Item | License | Fine-tune allowed? | Deploy allowed? | Notes |
|---|---|---|---|---|
| Indra weights v3.1.1 | AI PUBS Open RAIL-M (modified) | YES, explicit (§II ¶36) | YES (restrictions = Attachment A misuse list; internal OCR unaffected) | Derivative weights → same license + notice on redistribution |
indra_ocr code |
Apache-2.0 | n/a | YES | |
| PaddleOCR (baseline) | Apache-2.0 | YES (we already fine-tune it) | YES | current deployed |
| PP-OCRv5 Mobile Devanagari weights | Apache-2.0 | YES | YES | deployed |
| Nepali train corpus | per docs/DATA_LICENSING.md [B] |
pending review | — | no removal of datasets from remote |
| Frozen eval sets | per docs/DATA_LICENSING.md [B] |
evaluation only | — | |
| Noto Sans Devanagari (testrender only) | OFL 1.1 | YES (fonts embedding fine) | test asset only | tools/indra_eval/testdata/ |
§14 Deployment comparison
| Axis | Indra v3.1.1 (rec-only swap / full) | PP-OCRv5 Mobile Devanagari deployed |
|---|---|---|
| Recognizer params | 1.236 M (verified) | rec ~similar class (Mobile range; exact count from remote bench) |
| Runtime | onnxruntime CPU (already installed here) | paddle (remote) |
| Extra machinery needed | none — single ONNX + meta.json vocab JSON | paddle + FineTuned assets + converter glue |
| Latency (lines/s, T4 CPU) | measure (A/D/E) | measure |
| Deterministic CPU memory | flat batch (mask) | paddle per-page buffers |
| Order/reading model | included (order.onnx; optional) | not used for plain OCR flow |
| Honest staffing: fine-tune effort | reimplement trainer (~1–2 days) | already exists for Paddle |
| Dependency risk | pinned HF revision + local files | paddle version drift |
§17 Decision tree
- Run A (zero-shot rec-lines + full) on the frozen sets.
- If Indra rec CER < deployed rec CER on 2 of 3 sets: proceed to the hybrid experiment D (PP-det + Indra-rec); if D holds the win, plan a rec-swap deployment trial alongside the existing PP pipeline(1 week).
- If Indra rec CER ≥ deployed rec CER: record the gap per slice. If the gap is train-material (conjuncts/₹/digits): run B (fine-tune rec from safetensors, ~10–20k steps); expected to close a distribution-gap-shaped loss; decide by ΔCER on the eval sets.
- Regardless of A's outcome, the graceful fallback stands: bringing Indra recognizer into our PP-OCR det pipeline is a drop-in component (same crops→ink-map contract), so nothing about A/B/D commits us.
- Det/order fine-tuning (C) stays out of scope until det-on-real-scan
failure is demonstrated in A-full (
det recall < line recall of deployed). - Never touch: datasets on the remote, the deployed pipeline's recipes,
its eval scripts, or
~/.ssh/~/.bashrc. All experiments run in a separate directory (~/ocr/OCR_INDRA/), copying only code/weights in.
§18 Status summary
- [V] runnable here end-to-end: smoke chain (synth page → rec-only → full → CER), all inputs/outputs/paths in Investigation §8.
- [B] everything requiring
khalti-t4: A.2 (frozen sets), B data, C skip decision input, licensing rows, E throughput on deployment hardware. - [V] fine-tune toolkit verified offline 2026-10-20 (
tools/exp/verify_bundle.sh: export exactness pass; mechanics training + post-train gate pass; A/B orchestrations complete on a local stand-in set); int8 static/dynamic behaviour probed (dynamic = slower trap, static = ~1.4× faster, 3.5× smaller). - Open: WP det recall numbers, paddle latency comparison, int8 CER on real calibration + deployment-box throughput.