ocr-projects-archive / OCR_INDRA /docs /INDRA_INVESTIGATION.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
17.6 kB

Indra OCR β€” Engineering Investigation (v3.1.1, package 0.3.2)

Investigator: OCR_INDRA workspace, 2026-10-17. Method: every claim below is backed by (a) model artifacts inspected bit-for-bit (models/indra-ocr-v3.1.1/, Hub revision v3.1.1) or (b) source at a cited file:line in the cloned repo (indra/, commit 3d773ae, tag v0.3.2). Verification scripts live in tools/indra_eval/ and are re-runnable.

Status legend: [V] verified with evidence here, [B] blocked (needs the remote machine khalti-t4), [X] closed as refuted/inaccurate vs the earlier report INDRA_OCR_NEPALI_FEASIBILITY.md.


0. Verdict

The recognizer is trainable in principle and Nepali-ready in vocabulary, but fine-tuning is a reimplementation project: the repo ships no training code and no training weights for anything but the recognizer.

  • Zero-shot evaluation (Experiment A) is ready today β€” the adapter (tools/indra_eval/eval_indra.py) runs Indra end-to-end on CPU and scores CER; it needs only the frozen Nepali eval sets from the remote machine [B].
  • The CTC recognizer (rec.onnx) covers 100% of Devanagari (all 128 assigned codepoints U+0900–U+097F verified in the vocab) β€” no head surgery needed [V].
  • In a same-crop head-to-head under our environment, the ~1.24 M-param Indra recognizer scored CER 3.49% on a synthetic Nepali page vs the deployed PP-OCRv5-Mobile Devanagari recognizer's published behavior on similar synthetic material (to be re-measured on identical crops) [V smoke / B].
  • Fine-tuning only the recognizer with model.safetensors as init is the only supported adaptation path: the artifacts ship trainable weights for rec/ exclusively; det/order/orient/math exist only as ONNX graphs with no accompanying state dict [V].

1. What exactly is the ~1M-param recognizer? [V]

Answer: a MobileNetV3-style 1-D CNN trunk with padding-masked SE blocks and a flat CTC head β€” 1,236,366 parameters exactly, no RNN, no attention.

Evidence: tools/indra_eval/inspect_rec_nodes.py (201-node dump of rec.onnx), tools/indra_eval/inspect_rec_mask.py (intermediate-tensor run), tools/indra_eval/inspect_models.py (safetensors ↔ ONNX initializer cross-check: 108 tensors, 1,236,366 F32 params, byte-identical set).

1.1 Tensor flow (all verified from the graph)

input  x      [B,1,32,W]  f32 ink map   (0=paper, 1=ink; see Β§1.3)
input  widths [B]         int64         (true crop widths BEFORE padding)

0  Conv(k=32x4, stride=32x4, 1β†’128, pad 0)   "collapse height, stride time 4"
2  Squeeze axis 2 β†’ [B,128,T]  with T = W/4
1  BN(eps 1e-5, stem-only BatchNormalization) β†’ HardSwish
43 Mul Γ— Cast_4  (padding mask, see Β§1.2)  β†’ trunk input

10 Γ— { dw Conv [128,1,9] g=128 pad 4/4
       expand Conv 1Γ—1 128β†’256 β†’ HardSwish
       SE: Mul(mask) β†’ ReduceSum(t)  ←/β†’ ReduceSum(mask) = valid count
           Div (mean over VALID timesteps, Clip min 1)
           fc1 256β†’64 β†’ Relu β†’ fc2 64β†’256 β†’ Sigmoid β†’ gate Mul
       project Conv 1Γ—1 256β†’128
       Add residual β†’ Mul(mask)  }          ← mask re-applied per block

188 Conv 1Γ—1 128β†’1678 (head) β†’ Transpose [B,T,1678]
191 ArgMax β†’ ids [B,T]
190/193..200 softmax-max (logsumexp-free: ReduceMax, Exp, ReduceSum, Recip)
           β†’ conf [B,T]
27  Clip(lengths) β†’ lengths [B]

Op census (from the node dump): 52 Conv, 11 HardSwish, 10 Relu, 10 Sigmoid (SE gates), 32 Mul, 12 ReduceSum, 1 BatchNormalization, 1 ArgMax β€” plus glue. The parameters: 10 identical residual blocks (dw 128Γ—1Γ—9 + bias, expand 256Γ—128, project 128Γ—256, SE 64Γ—256 + 256Γ—64) β‰ˆ 118 k/block + stem 128Γ—32Γ—4 + BN 4Γ—128 + head 1678Γ—128 β‰ˆ 1,236,366 total [V].

1.2 The padding mask Cast_4 β€” semantics now fully pinned [V, closes a [U]]

Cast_4 = (arange(W/4) < lengths).float() of shape [B,1,W'], built in-graph from:

  • nodes 3–27: lengths = floor((widths+3)/4) = ceil(widths/4), clipped to W/4 (Clip max = Shape(x)[2] / 4... clipped to the feature length); numeric table: widths 1,2,3,4 β†’ 1; 5 β†’ 2; 9 β†’ 3; 65 β†’ 17; 129 β†’ 32 (= W/4, clipped); 200 β†’ 32.
  • nodes 31–40: Range(0, W/4) < lengths (broadcast [1,W'] < [B,1]) β†’ Cast f32 twice (PyTorch export artifact of unsqueeze().to(f32)).

The mask is consumed three ways (this closes the earlier "[U] what does Cast_4 do?"):

  1. post-stem zeroing (node 43) and per-block output re-zeroing (nodes 43, 61, 75, …, 187): the trunk never carries energy into padded timesteps β€” proven numerically: with ink drawn across the whole 128-column input but widths=64, final-block features at t β‰₯ 16 are exactly 0.0 (max |f| = 0.0) while the same pixels unmasked (widths=128) reach |f|=15.6;
  2. padding-aware SE pooling (each block: ReduceSum(HW Γ— mask) / max(ReduceSum(mask), 1) β†’ channel means over VALID timesteps only; the mask sum is the divisor, so padded columns cannot dilute the squeeze stats);
  3. the head reads masked features (node 187β†’188), so at padded timesteps the graph emits whatever the head computes on a zero vector β€” measured: argmax = 0 (blank) with conf β‰ˆ 0.78. ids/conf at t β‰₯ lengths are NOT special-cased; callers must truncate by lengths. pipeline.py:297 does exactly this (n = int(lengths[k])).

1.3 Time stride is 4, not 8 [X β€” closes a wrong claim]

rec_preproc.py:24 says WIDTH_MULTIPLE = 8 # widths are padded to a multiple of this (timestep stride). The graph contradicts the parenthetical: the stem Conv strides are [32,4], giving one timestep per 4 input columns (T = W/4, lengths = ceil(widths/4)), and padding to a multiple of 8 is about the W dimension of the input only (it makes every 2 timesteps aligned, a vectorization nicety). Verified numerically: T = 32 for W = 128, and lengths equals ceil(widths/4) at every measured value. The comment is wrong as written; the code is right.

1.4 Output contract [V β€” closes a [U]]

Graph outputs are exactly ids [B,T] int64, conf [B,T] f32, lengths [B] int64 (argmax + softmax max prob + valid count computed in-graph). pipeline.py:294 consumes them positionally as best, pmax, lengths, then:

  • truncates to n = int(lengths[k]) (:297),
  • CTC-greedy collapse: keep β‰  blank(0) and skip repeats (:299),
  • confidence = mean pmax over kept tokens (:315),
  • drops the whole line if conf < MIN_CONF = 0.5 (:60, :316–319).

Any retrained recognizer exported to ONNX must reproduce this contract (argmax + softmax-max + lengths on the same output names/order), or pipeline.py:294 and the x-positions step (which maps CTC peak timesteps back to page x via the crop scale, :295–306) must be updated.

1.5 Recognizer input preprocessing (the shard that fine-tuning must clone)

rec_preproc.py β€” crop_line (:27): expand box by PAD_Y=0.2 Γ— h vertically and PAD_X=0.3 Γ— h horizontally (:22–23), restrict to page, resize so height = LINE_HEIGHT = 32 keeping aspect (out_w = max(WIDTH_MULTIPLE, min(max_width=4096, round(cw*32/ch)))), producing an ink map where paper = 0. pad_batch (:56) right-pads W to a multiple of 8 in the batch; width_batches (:69) groups crops so W-dim batch padding is bounded. The TRAINER uses the same crop per its docstring (:5–10 "expand … resize … ink map") β€” the docstring states train/infer share this shard, which is why zero-shot works at all and why fine-tuning must reimplement crop_line + pad_batch bit-exactly.

2. What handles Devanagari? [V]

Only rec.onnx reads Devanagari β€” and its vocabulary already includes every assigned Devanagari codepoint.

  • vocab.py:170 Vocab.multi() β†’ 1,677 tokens + blank class 0 (:3, :207), head = 1,678 classes. The meta.json vocab string is exactly this vocab (tools/indra_eval/inspect_vocab.py).
  • Coverage: 128/128 assigned codepoints in U+0900–U+097F β€” 80 Lo letters, 11 Mc vowel signs, 23 Mn marks incl. ΰ€Ό (U+093C nukta) and the conjunct virama ΰ€•ΰ₯, plus ΰ₯‰ (U+0950), ΰ€ ΰ₯€ (U+0964), ΰ₯₯ (U+0965), digits ΰ₯¦β€“ΰ₯― (U+0966–096F), β‚Ή (U+20B9). Bengali block also present (96 codepoints).
  • The recognizer is routed by two fixed guards, not the network:
    • script.onnx (hanzi binary classifier, sigmoid scalar; pipeline.py:274–289) sends hanzi lines to rec_cjk.onnx β€” irrelevant for Nepali pages (all-Devanagari pages never trigger it; the classifier is line-level and trained on hanzi).
    • vocab.canonical (:42) NFC-normalizes; Vocab.encode (:212) drops out-of-vocab codepoints β€” fine-tuning data must be pre-normalized the same way or the loss silently degrades (the trainer's data pipeline is closed; safest is to clone canonical/encode).
  • Confirmed counter-evidence collected in the smoke run: ΰ€Ό (nukta) is often dropped contextually at 32 px height (ΰ€°ΰ€Ό and ΰ€° differ only by a 1–2 px dot); β‚Ή was read as ΰ€°. Both are training-distribution issues, not capability gaps β€” exactly the kind of thing fine-tuning would fix (Experiment B).

3. Component map: trainable / fixed-but-replaceable / fixed [V]

Component Where weights come from Trainable via shipped artifacts? Replaceability
rec.onnx recognizer model.safetensors (108 tensors, all rec/) YES β€” the only shipped state dict; init for a torch CTC retrain (Experiment B) head/vocab identical β†’ drop-in for +1,678 classes
det.onnx line detector ONNX only NO state dict shipped could retrain in torch (targets spec exists: det_post.encode_targets, det_post.py:48 β€” core logit + top/bottom distances per line, CORE_FRAC=0.4, CORE_X_SHRINK=0.15) β€” but training code is closed β†’ high effort
order.onnx reading order/layout ONNX only NO same: order_post.label_order/decode_* reveal supervision (10 block classes) but no weights
orient.onnx page rotation ONNX only NO trivial to replace with a torch/tlclassifier or heuristic
mask.onnx CenterMask NOT SHIPPED in v3.1.1 (pipeline.py:201–205 optional) β€” absent here; crop_lines(gray, boxes, 32, None) runs w/o it (verified)
script.onnx + rec_cjk.onnx ONNX only NO optional; would be excluded from a Devanagari build
math*.onnx (4 files) NOT shipped unless math=True (models.py:22–44) β€” out of scope
geometry/deskew/layout postproc pure Python (pipeline.py:37–149, layout_post.py) fixed code, no training needed replaceable with own code paths
text-layer (PDF) naive path textlayer.py fixed code irrelevant to scans/Nepali notes

The decisive asymmetry: Google ships a recognizer-only training starting point. Fine-tuning anything else means either (a) training from scratch (det targets spec'd, but no reference weights/recipe) or (b) leaving det/order frozen ONNX and only exercising the rec head β€” that's Experiment B; both are documented in INDRA_EXPERIMENTS.md.

4. Integration architecture (how a page becomes text) [V]

pipeline.py:456–470 (OrtOCR.__call__):

  1. upright(): optional orient.onnx turn above ORIENT_MIN = 0.95 (:54) + skew estimate/deskew up to 5Β° (:62–63).
  2. detect(): det.onnx on a page resized long-side to detector input, threshold β†’ line boxes (uses det_post.page_to_input / decode; ADAPTIVE_MIN_H=9px re-detect pass at a larger scale, :37–40).
  3. recognize() (:264–320): CenterMask if present (not shipped); per-line crop_line β†’ pad_batch β†’ rec session, CTC collapse & conf gating (Β§1.2/Β§1.4).
  4. line splitting at wide gaps (SPLIT_GAP=3.0, CELL_GAP=1.8, GUTTER_GAP=1.6, :44–47) and gutter detection (:342).
  5. reading_order() on order.onnx (feats [1,N,16], outputs pairwise salience a, block class cls [10 classes], same-block same, table t-heads tab) then layout (layout_post.py: page_blocks, page_markdown).

MIN_CONF=0.5 means a busy page with low-quality Devanagari can silently emit empty lines in Experiment A β€” compare-on-GT-boxes (rec-lines mode) sidesteps this entirely, which is why Experiment A does both full-pipeline and rec-lines scoring.

5. Licensing [V]

  • Model weights (models/indra-ocr-v3.1.1/LICENSE): AI PUBS OPEN RAIL-M (MODIFIED), v0.1, 2023-03-02. Section II ΒΆ36 states verbatim: "Use may include creating any content with, fine-tuning, updating, running, training, evaluating and/or re-parametrizing the Model" β€” fine-tuning is explicitly licensed. ΒΆ5 attaches Attachment A use-based restrictions (standard OpenRAIL misuse list β€” weapons/illegal/defamation; OCR and document processing is untouched) and requires passing the same restrictions to distributees; ΒΆ4 permits hosting/redistribution with notice. Derivative weights (fine-tuned checkpoints, distillations) fall under the same license.
  • Code (indra/LICENSE): Apache-2.0 β€” reuse, modification, redistribution of the inference package are standard.
  • Practical consequence: our fine-tuned recognizer (Experiment B) is legally a "Derivative of the Model" under OpenRAIL-M β€” internal deployment and evaluation are unrestricted; redistribution requires keeping the license + restriction notice. Full matrix incl. Nepali datasets: Β§12 of INDRA_EXPERIMENTS.md (dataset side [B] β€” needs docs/DATA_LICENSING.md on the remote machine).

6. Closing the prior report's [U] items

Prior unknown Status Evidence
exact recognizer arch (RNN? CNN?) [V] + [X]: pure CNN+CTC, NO RNN/attention Β§1.1, node dump
timestep stride [X]: 4, not 8 Β§1.3 (graph constants + inc.timings)
output contract (best/pmax docstrings vs solo head) [V]: ids,conf,lengths names; truncation mandatory Β§1.4
safetensors ↔ ONNX mapping (esp. depthwise onnx::Conv_369…397) [V]: 108 tensors = 108 initializers, 1,236,366 params both ways; dw weights are rec/onnx::Conv_369 [128,1,9] + …370 [128] bias pairs inspect_models.py
Devanagari coverage claim [V]: 128/128 codepoints in vocab inspect_vocab.py
What re-export must reproduce [V]: mask + per-block re-zeroing + SE-valid-norm + decode tail; lengths = ceil(widths/4) clip W/4 Β§1.2, inspect_rec_mask.py
Confidence semantics of padded steps [X]: head-on-zeros β‰ˆ blank conf 0.78; caller truncates Β§1.2(3)
vocab[token i+1] ↔ blank 0 [V] vocab.py:3,207,241–242
rec input normalization [V]: ink map only (0 paper, 1 ink) per crop_line, no mean/std rec_preproc.py:1–10
enables fine-tune? [V] yes for rec only (weights exist; no training code; CTC contract reconstructed) Β§3, Β§B doc
full pipeline runs on Nepali test image [V] smoke: synthetic shaped page, 10/10 lines, CER 2.6–3.5% results/smoke/
what differs in real (non-synth) Nepali [B] frozen remote sets + baseline Experiment A
trained det/order weights for fine-tune [V] not shipped (ONNX only) Β§3

7. Open items (not closed by this investigation)

  1. [B] Zero-shot numbers on the frozen Nepali eval sets (needs khalti-t4).
  2. [B] Head-to-head vs PP-OCRv5 Mobile Devanagari on the same crops β€” the remote machine holds the paddle recognizer + eval harness. The local smoke CER of 2.6–3.5% is on a synthetic page with a 32 px font and is not a benchmark.
  3. [B] Licensing matrix final values need docs/DATA_LICENSING.md and docs/DEVANAGARI_RESOURCES_INVENTORY.md (remote).
  4. FX-detector quality on Devanagari (qubee endpoints, aggressively slanted conjuncts) is unknowable without the det model being exercised on real scans β€” that's part of Experiment A full-pipeline.
  5. The root training repo (referenced by name in the closed code) is not accessible; Archived recipe unknowns: batch sizes, LR schedule, data mix.

8. Reproduce everything

cd /mnt/c/Users/Dell/Desktop/OCR_INDRA            # or wherever the repo is
PY=/tmp/opencode/indra-lib                        # pip --target deps
export PYTHONPATH="$PWD/indra:$PY"

python3 tools/indra_eval/inspect_vocab.py          # vocab + Devanagari coverage
python3 tools/indra_eval/inspect_models.py         # graph I/O + param count
python3 tools/indra_eval/inspect_rec_nodes.py      # full node dump + smoke run
python3 tools/indra_eval/inspect_rec_mask.py       # mask semantics + lengths table

python3 tools/indra_eval/make_synth_page.py results/smoke   # synth page + GT
python3 tools/indra_eval/eval_indra.py predict results/smoke/synth_nepali_page.png \
        --boxes results/smoke/synth_nepali_gt.jsonl --jsonl results/smoke/rec_only.jsonl
python3 tools/indra_eval/eval_indra.py predict results/smoke/synth_nepali_page.png \
        --jsonl results/smoke/full_run.jsonl --print-text
python3 tools/indra_eval/eval_indra.py cer results/smoke/synth_nepali_record.jsonl \
        results/smoke/rec_only.jsonl --detail results/smoke/cer_rec_only.json --rec-only
python3 tools/indra_eval/eval_indra.py cer results/smoke/synth_nepali_record.jsonl \
        results/smoke/full_run.jsonl --detail results/smoke/cer_full.json

Observed (this machine, WSL, CPU, 8 cores, 3 GB RAM): rec-only CER 3.49% / WER 15.79%; full-pipeline CER 2.62% / WER 13.16%; 10/10 lines found; mean_conf 0.962–0.963. Timings on this box are meaningless (det β‰ˆ 18.6 min β€” machine-starved; bench on the T4).