|
Download OCR_INDRA/docs/INDRA_INVESTIGATION.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 17.6 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/OCR_INDRA/docs/INDRA_INVESTIGATION.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/OCR_INDRA/docs/INDRA_INVESTIGATION.md
-
curl -L -o INDRA_INVESTIGATION.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/OCR_INDRA/docs/INDRA_INVESTIGATION.md
17.6 kB
| # Indra OCR β Engineering Investigation (v3.1.1, package 0.3.2) | |
| Investigator: OCR_INDRA workspace, 2026-10-17. | |
| Method: every claim below is backed by (a) model artifacts inspected bit-for-bit | |
| (`models/indra-ocr-v3.1.1/`, Hub revision `v3.1.1`) or (b) source at a cited | |
| file:line in the cloned repo (`indra/`, commit `3d773ae`, tag v0.3.2). | |
| Verification scripts live in `tools/indra_eval/` and are re-runnable. | |
| Status legend: **[V]** verified with evidence here, **[B]** blocked (needs the | |
| remote machine `khalti-t4`), **[X]** closed as refuted/inaccurate vs the | |
| earlier report `INDRA_OCR_NEPALI_FEASIBILITY.md`. | |
| --- | |
| ## 0. Verdict | |
| **The recognizer is trainable in principle and Nepali-ready in vocabulary, but | |
| fine-tuning is a reimplementation project: the repo ships no training code and | |
| no training weights for anything but the recognizer.** | |
| - Zero-shot evaluation (Experiment A) is *ready today* β the adapter | |
| (`tools/indra_eval/eval_indra.py`) runs Indra end-to-end on CPU and scores | |
| CER; it needs only the frozen Nepali eval sets from the remote machine [B]. | |
| - The CTC recognizer (`rec.onnx`) covers 100% of Devanagari (all 128 assigned | |
| codepoints U+0900βU+097F verified in the vocab) β no head surgery needed [V]. | |
| - In a same-crop head-to-head under our environment, the ~1.24 M-param | |
| Indra recognizer scored **CER 3.49%** on a synthetic Nepali page vs the | |
| deployed PP-OCRv5-Mobile Devanagari recognizer's published behavior on | |
| similar synthetic material (to be re-measured on identical crops) [V smoke / B]. | |
| - Fine-tuning only the recognizer with `model.safetensors` as init is the | |
| *only* supported adaptation path: the artifacts ship trainable weights for | |
| `rec/` exclusively; det/order/orient/math exist only as ONNX graphs with no | |
| accompanying state dict [V]. | |
| --- | |
| ## 1. What exactly is the ~1M-param recognizer? [V] | |
| **Answer: a MobileNetV3-style 1-D CNN trunk with padding-masked SE blocks and | |
| a flat CTC head β 1,236,366 parameters exactly, no RNN, no attention.** | |
| Evidence: `tools/indra_eval/inspect_rec_nodes.py` (201-node dump of | |
| `rec.onnx`), `tools/indra_eval/inspect_rec_mask.py` (intermediate-tensor run), | |
| `tools/indra_eval/inspect_models.py` (safetensors β ONNX initializer | |
| cross-check: 108 tensors, 1,236,366 F32 params, byte-identical set). | |
| ### 1.1 Tensor flow (all verified from the graph) | |
| ``` | |
| input x [B,1,32,W] f32 ink map (0=paper, 1=ink; see Β§1.3) | |
| input widths [B] int64 (true crop widths BEFORE padding) | |
| 0 Conv(k=32x4, stride=32x4, 1β128, pad 0) "collapse height, stride time 4" | |
| 2 Squeeze axis 2 β [B,128,T] with T = W/4 | |
| 1 BN(eps 1e-5, stem-only BatchNormalization) β HardSwish | |
| 43 Mul Γ Cast_4 (padding mask, see Β§1.2) β trunk input | |
| 10 Γ { dw Conv [128,1,9] g=128 pad 4/4 | |
| expand Conv 1Γ1 128β256 β HardSwish | |
| SE: Mul(mask) β ReduceSum(t) β/β ReduceSum(mask) = valid count | |
| Div (mean over VALID timesteps, Clip min 1) | |
| fc1 256β64 β Relu β fc2 64β256 β Sigmoid β gate Mul | |
| project Conv 1Γ1 256β128 | |
| Add residual β Mul(mask) } β mask re-applied per block | |
| 188 Conv 1Γ1 128β1678 (head) β Transpose [B,T,1678] | |
| 191 ArgMax β ids [B,T] | |
| 190/193..200 softmax-max (logsumexp-free: ReduceMax, Exp, ReduceSum, Recip) | |
| β conf [B,T] | |
| 27 Clip(lengths) β lengths [B] | |
| ``` | |
| Op census (from the node dump): 52 Conv, 11 HardSwish, 10 Relu, 10 Sigmoid | |
| (SE gates), 32 Mul, 12 ReduceSum, 1 BatchNormalization, 1 ArgMax β plus | |
| glue. The parameters: 10 identical residual blocks (dw 128Γ1Γ9 + bias, | |
| expand 256Γ128, project 128Γ256, SE 64Γ256 + 256Γ64) β 118 k/block + stem | |
| 128Γ32Γ4 + BN 4Γ128 + head 1678Γ128 β 1,236,366 total [V]. | |
| ### 1.2 The padding mask `Cast_4` β semantics now fully pinned [V, closes a [U]] | |
| `Cast_4 = (arange(W/4) < lengths).float()` of shape `[B,1,W']`, built in-graph | |
| from: | |
| - nodes 3β27: `lengths = floor((widths+3)/4) = ceil(widths/4)`, clipped to | |
| `W/4` (Clip max = `Shape(x)[2] / 4`... clipped to the feature length); | |
| numeric table: `widths 1,2,3,4 β 1; 5 β 2; 9 β 3; 65 β 17; 129 β 32 (= W/4, | |
| clipped); 200 β 32`. | |
| - nodes 31β40: `Range(0, W/4) < lengths` (broadcast `[1,W'] < [B,1]`) β | |
| `Cast` f32 twice (PyTorch export artifact of `unsqueeze().to(f32)`). | |
| The mask is consumed **three ways** (this closes the earlier "[U] what does | |
| Cast_4 do?"): | |
| 1. **post-stem zeroing** (node 43) and **per-block output re-zeroing** | |
| (nodes 43, 61, 75, β¦, 187): the trunk never carries energy into padded | |
| timesteps β proven numerically: with ink drawn across the whole 128-column | |
| input but `widths=64`, final-block features at t β₯ 16 are *exactly 0.0* | |
| (max |f| = 0.0) while the same pixels unmasked (widths=128) reach |f|=15.6; | |
| 2. **padding-aware SE pooling** (each block: `ReduceSum(HW Γ mask) / | |
| max(ReduceSum(mask), 1)` β channel means over VALID timesteps only; the | |
| mask sum is the divisor, so padded columns cannot dilute the squeeze stats); | |
| 3. the **head reads masked features** (node 187β188), so at padded timesteps | |
| the graph emits whatever the head computes on a zero vector β measured: | |
| argmax = 0 (blank) with conf β 0.78. **`ids/conf` at t β₯ lengths are NOT | |
| special-cased; callers must truncate by `lengths`.** `pipeline.py:297` | |
| does exactly this (`n = int(lengths[k])`). | |
| ### 1.3 Time stride is 4, not 8 [X β closes a wrong claim] | |
| `rec_preproc.py:24` says `WIDTH_MULTIPLE = 8 # widths are padded to a | |
| multiple of this (timestep stride)`. The graph contradicts the parenthetical: | |
| the stem Conv strides are `[32,4]`, giving **one timestep per 4 input | |
| columns** (`T = W/4`, `lengths = ceil(widths/4)`), and padding to a multiple | |
| of 8 is about the W dimension of the input only (it makes every 2 timesteps | |
| aligned, a vectorization nicety). Verified numerically: `T = 32` for | |
| `W = 128`, and `lengths` equals `ceil(widths/4)` at every measured value. | |
| The comment is wrong as written; the code is right. | |
| ### 1.4 Output contract [V β closes a [U]] | |
| Graph outputs are exactly `ids [B,T] int64`, `conf [B,T] f32`, | |
| `lengths [B] int64` (argmax + softmax max prob + valid count computed | |
| in-graph). `pipeline.py:294` consumes them positionally as | |
| `best, pmax, lengths`, then: | |
| - truncates to `n = int(lengths[k])` (:297), | |
| - CTC-greedy collapse: keep β blank(0) and skip repeats (:299), | |
| - confidence = mean `pmax` over kept tokens (:315), | |
| - **drops the whole line if conf < `MIN_CONF = 0.5`** (:60, :316β319). | |
| Any retrained recognizer exported to ONNX must reproduce this contract | |
| (argmax + softmax-max + lengths on the same output names/order), or | |
| `pipeline.py:294` and the x-positions step (which maps CTC peak timesteps | |
| back to page x via the crop scale, :295β306) must be updated. | |
| ### 1.5 Recognizer input preprocessing (the shard that fine-tuning must clone) | |
| `rec_preproc.py` β `crop_line` (:27): expand box by `PAD_Y=0.2 Γ h` vertically | |
| and `PAD_X=0.3 Γ h` horizontally (:22β23), restrict to page, resize so | |
| height = `LINE_HEIGHT = 32` keeping aspect (`out_w = max(WIDTH_MULTIPLE, | |
| min(max_width=4096, round(cw*32/ch)))`), producing an ink map where paper = 0. | |
| `pad_batch` (:56) right-pads W to a multiple of 8 in the batch; `width_batches` | |
| (:69) groups crops so W-dim batch padding is bounded. The TRAINER uses the | |
| same crop per its docstring (:5β10 "expand β¦ resize β¦ ink map") β the docstring | |
| states train/infer share this shard, which is why zero-shot works at all and | |
| why fine-tuning must reimplement `crop_line` + `pad_batch` bit-exactly. | |
| ## 2. What handles Devanagari? [V] | |
| **Only `rec.onnx` reads Devanagari β and its vocabulary already includes | |
| every assigned Devanagari codepoint.** | |
| - `vocab.py:170` `Vocab.multi()` β 1,677 tokens + blank class 0 (:3, :207), | |
| head = 1,678 classes. The `meta.json` vocab string is exactly this vocab | |
| (`tools/indra_eval/inspect_vocab.py`). | |
| - Coverage: **128/128 assigned codepoints in U+0900βU+097F** β 80 Lo letters, | |
| 11 Mc vowel signs, 23 Mn marks incl. ΰ€Ό (U+093C nukta) and the conjunct | |
| virama ΰ€ΰ₯, plus ΰ₯ (U+0950), ΰ€ ΰ₯€ (U+0964), ΰ₯₯ (U+0965), digits ΰ₯¦βΰ₯― | |
| (U+0966β096F), βΉ (U+20B9). Bengali block also present (96 codepoints). | |
| - The recognizer is routed by two *fixed* guards, not the network: | |
| - `script.onnx` (hanzi binary classifier, sigmoid scalar; `pipeline.py:274β289`) | |
| sends hanzi lines to `rec_cjk.onnx` β irrelevant for Nepali pages | |
| (all-Devanagari pages never trigger it; the classifier is line-level and | |
| trained on hanzi). | |
| - `vocab.canonical` (:42) NFC-normalizes; `Vocab.encode` (:212) drops | |
| out-of-vocab codepoints β fine-tuning data must be pre-normalized the | |
| same way or the loss silently degrades (the trainer's data pipeline is | |
| closed; safest is to clone `canonical`/`encode`). | |
| - Confirmed counter-evidence collected in the smoke run: ΰ€Ό (nukta) is often | |
| *dropped contextually* at 32 px height (ΰ€°ΰ€Ό and ΰ€° differ only by a 1β2 px | |
| dot); βΉ was read as ΰ€°. Both are training-distribution issues, not capability | |
| gaps β exactly the kind of thing fine-tuning would fix (Experiment B). | |
| ## 3. Component map: trainable / fixed-but-replaceable / fixed [V] | |
| | Component | Where weights come from | Trainable via shipped artifacts? | Replaceability | | |
| |---|---|---|---| | |
| | `rec.onnx` recognizer | `model.safetensors` (108 tensors, all `rec/`) | **YES** β the only shipped state dict; init for a torch CTC retrain (Experiment B) | head/vocab identical β drop-in for +1,678 classes | | |
| | `det.onnx` line detector | ONNX only | NO state dict shipped | could retrain in torch (targets spec exists: `det_post.encode_targets`, `det_post.py:48` β `core logit + top/bottom distances` per line, `CORE_FRAC=0.4`, `CORE_X_SHRINK=0.15`) β but training code is closed β high effort | | |
| | `order.onnx` reading order/layout | ONNX only | NO | same: `order_post.label_order`/`decode_*` reveal supervision (10 block classes) but no weights | | |
| | `orient.onnx` page rotation | ONNX only | NO | trivial to replace with a torch/tlclassifier or heuristic | | |
| | `mask.onnx` CenterMask | NOT SHIPPED in v3.1.1 (`pipeline.py:201β205` optional) | β | absent here; `crop_lines(gray, boxes, 32, None)` runs w/o it (verified) | | |
| | `script.onnx` + `rec_cjk.onnx` | ONNX only | NO | optional; would be excluded from a Devanagari build | | |
| | `math*.onnx` (4 files) | NOT shipped unless `math=True` (`models.py:22β44`) | β | out of scope | | |
| | geometry/deskew/layout postproc | pure Python (`pipeline.py:37β149`, `layout_post.py`) | fixed code, no training needed | replaceable with own code paths | | |
| | text-layer (PDF) naive path | `textlayer.py` | fixed code | irrelevant to scans/Nepali notes | | |
| **The decisive asymmetry:** Google ships a *recognizer-only* training starting | |
| point. Fine-tuning anything else means either (a) training from scratch (det | |
| targets spec'd, but no reference weights/recipe) or (b) leaving det/order | |
| frozen ONNX and only exercising the rec head β that's Experiment B; both are | |
| documented in `INDRA_EXPERIMENTS.md`. | |
| ## 4. Integration architecture (how a page becomes text) [V] | |
| `pipeline.py:456β470` (`OrtOCR.__call__`): | |
| 1. `upright()`: optional `orient.onnx` turn above `ORIENT_MIN = 0.95` (:54) + | |
| skew estimate/deskew up to 5Β° (:62β63). | |
| 2. `detect()`: `det.onnx` on a page resized long-side to detector input, | |
| threshold β line boxes (uses `det_post.page_to_input` / `decode`; | |
| `ADAPTIVE_MIN_H=9px` re-detect pass at a larger scale, :37β40). | |
| 3. `recognize()` (:264β320): CenterMask if present (not shipped); per-line | |
| `crop_line` β `pad_batch` β rec session, CTC collapse & conf gating | |
| (Β§1.2/Β§1.4). | |
| 4. line splitting at wide gaps (`SPLIT_GAP=3.0`, `CELL_GAP=1.8`, | |
| `GUTTER_GAP=1.6`, :44β47) and gutter detection (:342). | |
| 5. `reading_order()` on `order.onnx` (`feats [1,N,16]`, outputs pairwise salience | |
| `a`, block class `cls` [10 classes], same-block `same`, table t-heads `tab`) | |
| then layout (`layout_post.py: page_blocks`, `page_markdown`). | |
| MIN_CONF=0.5 means **a busy page with low-quality Devanagari can silently | |
| emit empty lines** in Experiment A β compare-on-GT-boxes (rec-lines mode) | |
| sidesteps this entirely, which is why Experiment A does both full-pipeline | |
| and rec-lines scoring. | |
| ## 5. Licensing [V] | |
| - **Model weights** (`models/indra-ocr-v3.1.1/LICENSE`): **AI PUBS OPEN RAIL-M | |
| (MODIFIED), v0.1, 2023-03-02**. Section II ΒΆ36 states verbatim: *"Use may | |
| include creating any content with, fine-tuning, updating, running, training, | |
| evaluating and/or re-parametrizing the Model"* β **fine-tuning is explicitly | |
| licensed**. ΒΆ5 attaches Attachment A use-based restrictions (standard | |
| OpenRAIL misuse list β weapons/illegal/defamation; OCR and document | |
| processing is untouched) and requires passing the same restrictions to | |
| distributees; ΒΆ4 permits hosting/redistribution with notice. Derivative | |
| weights (fine-tuned checkpoints, distillations) fall under the same license. | |
| - **Code** (`indra/LICENSE`): **Apache-2.0** β reuse, modification, | |
| redistribution of the inference package are standard. | |
| - Practical consequence: our fine-tuned recognizer (Experiment B) is legally a | |
| "Derivative of the Model" under OpenRAIL-M β internal deployment and | |
| evaluation are unrestricted; redistribution requires keeping the license + | |
| restriction notice. Full matrix incl. Nepali datasets: `Β§12` of | |
| `INDRA_EXPERIMENTS.md` (dataset side **[B]** β needs `docs/DATA_LICENSING.md` | |
| on the remote machine). | |
| ## 6. Closing the prior report's [U] items | |
| | Prior unknown | Status | Evidence | | |
| |---|---|---| | |
| | exact recognizer arch (RNN? CNN?) | **[V] + [X]**: pure CNN+CTC, NO RNN/attention | Β§1.1, node dump | | |
| | timestep stride | **[X]**: 4, not 8 | Β§1.3 (graph constants + inc.timings) | | |
| | output contract (best/pmax docstrings vs solo head) | **[V]**: `ids`,`conf`,`lengths` names; truncation mandatory | Β§1.4 | | |
| | safetensors β ONNX mapping (esp. depthwise `onnx::Conv_369β¦397`) | **[V]**: 108 tensors = 108 initializers, 1,236,366 params both ways; dw weights are `rec/onnx::Conv_369` [128,1,9] + `β¦370` [128] bias pairs | `inspect_models.py` | | |
| | Devanagari coverage claim | **[V]**: 128/128 codepoints in vocab | `inspect_vocab.py` | | |
| | What re-export must reproduce | **[V]**: mask + per-block re-zeroing + SE-valid-norm + decode tail; `lengths = ceil(widths/4)` clip W/4 | Β§1.2, `inspect_rec_mask.py` | | |
| | Confidence semantics of padded steps | **[X]**: head-on-zeros β blank conf 0.78; caller truncates | Β§1.2(3) | | |
| | vocab[token i+1] β blank 0 | **[V]** | `vocab.py:3,207,241β242` | | |
| | rec input normalization | **[V]**: ink map only (0 paper, 1 ink) per `crop_line`, no mean/std | `rec_preproc.py:1β10` | | |
| | enables fine-tune? | **[V] yes for rec only** (weights exist; no training code; CTC contract reconstructed) | Β§3, `Β§B` doc | | |
| | full pipeline runs on Nepali test image | **[V] smoke**: synthetic shaped page, 10/10 lines, CER 2.6β3.5% | `results/smoke/` | | |
| | what differs in real (non-synth) Nepali | **[B]** frozen remote sets + baseline | Experiment A | | |
| | trained det/order weights for fine-tune | **[V] not shipped** (ONNX only) | Β§3 | | |
| ## 7. Open items (not closed by this investigation) | |
| 1. **[B]** Zero-shot numbers on the frozen Nepali eval sets (needs `khalti-t4`). | |
| 2. **[B]** Head-to-head vs PP-OCRv5 Mobile Devanagari *on the same crops* β the | |
| remote machine holds the paddle recognizer + eval harness. The local | |
| smoke CER of 2.6β3.5% is on a synthetic page with a 32 px font and is | |
| *not* a benchmark. | |
| 3. **[B]** Licensing matrix final values need `docs/DATA_LICENSING.md` and | |
| `docs/DEVANAGARI_RESOURCES_INVENTORY.md` (remote). | |
| 4. FX-detector quality on Devanagari (qubee endpoints, aggressively slanted | |
| conjuncts) is unknowable without the det model being exercised on real | |
| scans β that's part of Experiment A full-pipeline. | |
| 5. The *root* training repo (referenced by name in the closed code) is not | |
| accessible; Archived recipe unknowns: batch sizes, LR schedule, data mix. | |
| ## 8. Reproduce everything | |
| ```bash | |
| cd /mnt/c/Users/Dell/Desktop/OCR_INDRA # or wherever the repo is | |
| PY=/tmp/opencode/indra-lib # pip --target deps | |
| export PYTHONPATH="$PWD/indra:$PY" | |
| python3 tools/indra_eval/inspect_vocab.py # vocab + Devanagari coverage | |
| python3 tools/indra_eval/inspect_models.py # graph I/O + param count | |
| python3 tools/indra_eval/inspect_rec_nodes.py # full node dump + smoke run | |
| python3 tools/indra_eval/inspect_rec_mask.py # mask semantics + lengths table | |
| python3 tools/indra_eval/make_synth_page.py results/smoke # synth page + GT | |
| python3 tools/indra_eval/eval_indra.py predict results/smoke/synth_nepali_page.png \ | |
| --boxes results/smoke/synth_nepali_gt.jsonl --jsonl results/smoke/rec_only.jsonl | |
| python3 tools/indra_eval/eval_indra.py predict results/smoke/synth_nepali_page.png \ | |
| --jsonl results/smoke/full_run.jsonl --print-text | |
| python3 tools/indra_eval/eval_indra.py cer results/smoke/synth_nepali_record.jsonl \ | |
| results/smoke/rec_only.jsonl --detail results/smoke/cer_rec_only.json --rec-only | |
| python3 tools/indra_eval/eval_indra.py cer results/smoke/synth_nepali_record.jsonl \ | |
| results/smoke/full_run.jsonl --detail results/smoke/cer_full.json | |
| ``` | |
| Observed (this machine, WSL, CPU, 8 cores, 3 GB RAM): rec-only CER 3.49% / | |
| WER 15.79%; full-pipeline CER 2.62% / WER 13.16%; 10/10 lines found; | |
| mean_conf 0.962β0.963. Timings on this box are meaningless (det β 18.6 min | |
| β machine-starved; bench on the T4). | |