ocr-projects-archive / OCR_INDRA /docs /INDRA_INVESTIGATION.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
17.6 kB
# Indra OCR β€” Engineering Investigation (v3.1.1, package 0.3.2)
Investigator: OCR_INDRA workspace, 2026-10-17.
Method: every claim below is backed by (a) model artifacts inspected bit-for-bit
(`models/indra-ocr-v3.1.1/`, Hub revision `v3.1.1`) or (b) source at a cited
file:line in the cloned repo (`indra/`, commit `3d773ae`, tag v0.3.2).
Verification scripts live in `tools/indra_eval/` and are re-runnable.
Status legend: **[V]** verified with evidence here, **[B]** blocked (needs the
remote machine `khalti-t4`), **[X]** closed as refuted/inaccurate vs the
earlier report `INDRA_OCR_NEPALI_FEASIBILITY.md`.
---
## 0. Verdict
**The recognizer is trainable in principle and Nepali-ready in vocabulary, but
fine-tuning is a reimplementation project: the repo ships no training code and
no training weights for anything but the recognizer.**
- Zero-shot evaluation (Experiment A) is *ready today* β€” the adapter
(`tools/indra_eval/eval_indra.py`) runs Indra end-to-end on CPU and scores
CER; it needs only the frozen Nepali eval sets from the remote machine [B].
- The CTC recognizer (`rec.onnx`) covers 100% of Devanagari (all 128 assigned
codepoints U+0900–U+097F verified in the vocab) β€” no head surgery needed [V].
- In a same-crop head-to-head under our environment, the ~1.24 M-param
Indra recognizer scored **CER 3.49%** on a synthetic Nepali page vs the
deployed PP-OCRv5-Mobile Devanagari recognizer's published behavior on
similar synthetic material (to be re-measured on identical crops) [V smoke / B].
- Fine-tuning only the recognizer with `model.safetensors` as init is the
*only* supported adaptation path: the artifacts ship trainable weights for
`rec/` exclusively; det/order/orient/math exist only as ONNX graphs with no
accompanying state dict [V].
---
## 1. What exactly is the ~1M-param recognizer? [V]
**Answer: a MobileNetV3-style 1-D CNN trunk with padding-masked SE blocks and
a flat CTC head β€” 1,236,366 parameters exactly, no RNN, no attention.**
Evidence: `tools/indra_eval/inspect_rec_nodes.py` (201-node dump of
`rec.onnx`), `tools/indra_eval/inspect_rec_mask.py` (intermediate-tensor run),
`tools/indra_eval/inspect_models.py` (safetensors ↔ ONNX initializer
cross-check: 108 tensors, 1,236,366 F32 params, byte-identical set).
### 1.1 Tensor flow (all verified from the graph)
```
input x [B,1,32,W] f32 ink map (0=paper, 1=ink; see Β§1.3)
input widths [B] int64 (true crop widths BEFORE padding)
0 Conv(k=32x4, stride=32x4, 1β†’128, pad 0) "collapse height, stride time 4"
2 Squeeze axis 2 β†’ [B,128,T] with T = W/4
1 BN(eps 1e-5, stem-only BatchNormalization) β†’ HardSwish
43 Mul Γ— Cast_4 (padding mask, see Β§1.2) β†’ trunk input
10 Γ— { dw Conv [128,1,9] g=128 pad 4/4
expand Conv 1Γ—1 128β†’256 β†’ HardSwish
SE: Mul(mask) β†’ ReduceSum(t) ←/β†’ ReduceSum(mask) = valid count
Div (mean over VALID timesteps, Clip min 1)
fc1 256β†’64 β†’ Relu β†’ fc2 64β†’256 β†’ Sigmoid β†’ gate Mul
project Conv 1Γ—1 256β†’128
Add residual β†’ Mul(mask) } ← mask re-applied per block
188 Conv 1Γ—1 128β†’1678 (head) β†’ Transpose [B,T,1678]
191 ArgMax β†’ ids [B,T]
190/193..200 softmax-max (logsumexp-free: ReduceMax, Exp, ReduceSum, Recip)
β†’ conf [B,T]
27 Clip(lengths) β†’ lengths [B]
```
Op census (from the node dump): 52 Conv, 11 HardSwish, 10 Relu, 10 Sigmoid
(SE gates), 32 Mul, 12 ReduceSum, 1 BatchNormalization, 1 ArgMax β€” plus
glue. The parameters: 10 identical residual blocks (dw 128Γ—1Γ—9 + bias,
expand 256Γ—128, project 128Γ—256, SE 64Γ—256 + 256Γ—64) β‰ˆ 118 k/block + stem
128Γ—32Γ—4 + BN 4Γ—128 + head 1678Γ—128 β‰ˆ 1,236,366 total [V].
### 1.2 The padding mask `Cast_4` β€” semantics now fully pinned [V, closes a [U]]
`Cast_4 = (arange(W/4) < lengths).float()` of shape `[B,1,W']`, built in-graph
from:
- nodes 3–27: `lengths = floor((widths+3)/4) = ceil(widths/4)`, clipped to
`W/4` (Clip max = `Shape(x)[2] / 4`... clipped to the feature length);
numeric table: `widths 1,2,3,4 β†’ 1; 5 β†’ 2; 9 β†’ 3; 65 β†’ 17; 129 β†’ 32 (= W/4,
clipped); 200 β†’ 32`.
- nodes 31–40: `Range(0, W/4) < lengths` (broadcast `[1,W'] < [B,1]`) β†’
`Cast` f32 twice (PyTorch export artifact of `unsqueeze().to(f32)`).
The mask is consumed **three ways** (this closes the earlier "[U] what does
Cast_4 do?"):
1. **post-stem zeroing** (node 43) and **per-block output re-zeroing**
(nodes 43, 61, 75, …, 187): the trunk never carries energy into padded
timesteps β€” proven numerically: with ink drawn across the whole 128-column
input but `widths=64`, final-block features at t β‰₯ 16 are *exactly 0.0*
(max |f| = 0.0) while the same pixels unmasked (widths=128) reach |f|=15.6;
2. **padding-aware SE pooling** (each block: `ReduceSum(HW Γ— mask) /
max(ReduceSum(mask), 1)` β†’ channel means over VALID timesteps only; the
mask sum is the divisor, so padded columns cannot dilute the squeeze stats);
3. the **head reads masked features** (node 187β†’188), so at padded timesteps
the graph emits whatever the head computes on a zero vector β€” measured:
argmax = 0 (blank) with conf β‰ˆ 0.78. **`ids/conf` at t β‰₯ lengths are NOT
special-cased; callers must truncate by `lengths`.** `pipeline.py:297`
does exactly this (`n = int(lengths[k])`).
### 1.3 Time stride is 4, not 8 [X β€” closes a wrong claim]
`rec_preproc.py:24` says `WIDTH_MULTIPLE = 8 # widths are padded to a
multiple of this (timestep stride)`. The graph contradicts the parenthetical:
the stem Conv strides are `[32,4]`, giving **one timestep per 4 input
columns** (`T = W/4`, `lengths = ceil(widths/4)`), and padding to a multiple
of 8 is about the W dimension of the input only (it makes every 2 timesteps
aligned, a vectorization nicety). Verified numerically: `T = 32` for
`W = 128`, and `lengths` equals `ceil(widths/4)` at every measured value.
The comment is wrong as written; the code is right.
### 1.4 Output contract [V β€” closes a [U]]
Graph outputs are exactly `ids [B,T] int64`, `conf [B,T] f32`,
`lengths [B] int64` (argmax + softmax max prob + valid count computed
in-graph). `pipeline.py:294` consumes them positionally as
`best, pmax, lengths`, then:
- truncates to `n = int(lengths[k])` (:297),
- CTC-greedy collapse: keep β‰  blank(0) and skip repeats (:299),
- confidence = mean `pmax` over kept tokens (:315),
- **drops the whole line if conf < `MIN_CONF = 0.5`** (:60, :316–319).
Any retrained recognizer exported to ONNX must reproduce this contract
(argmax + softmax-max + lengths on the same output names/order), or
`pipeline.py:294` and the x-positions step (which maps CTC peak timesteps
back to page x via the crop scale, :295–306) must be updated.
### 1.5 Recognizer input preprocessing (the shard that fine-tuning must clone)
`rec_preproc.py` β€” `crop_line` (:27): expand box by `PAD_Y=0.2 Γ— h` vertically
and `PAD_X=0.3 Γ— h` horizontally (:22–23), restrict to page, resize so
height = `LINE_HEIGHT = 32` keeping aspect (`out_w = max(WIDTH_MULTIPLE,
min(max_width=4096, round(cw*32/ch)))`), producing an ink map where paper = 0.
`pad_batch` (:56) right-pads W to a multiple of 8 in the batch; `width_batches`
(:69) groups crops so W-dim batch padding is bounded. The TRAINER uses the
same crop per its docstring (:5–10 "expand … resize … ink map") β€” the docstring
states train/infer share this shard, which is why zero-shot works at all and
why fine-tuning must reimplement `crop_line` + `pad_batch` bit-exactly.
## 2. What handles Devanagari? [V]
**Only `rec.onnx` reads Devanagari β€” and its vocabulary already includes
every assigned Devanagari codepoint.**
- `vocab.py:170` `Vocab.multi()` β†’ 1,677 tokens + blank class 0 (:3, :207),
head = 1,678 classes. The `meta.json` vocab string is exactly this vocab
(`tools/indra_eval/inspect_vocab.py`).
- Coverage: **128/128 assigned codepoints in U+0900–U+097F** β€” 80 Lo letters,
11 Mc vowel signs, 23 Mn marks incl. ΰ€Ό (U+093C nukta) and the conjunct
virama ΰ€•ΰ₯, plus ΰ₯‰ (U+0950), ΰ€ ΰ₯€ (U+0964), ΰ₯₯ (U+0965), digits ΰ₯¦β€“ΰ₯―
(U+0966–096F), β‚Ή (U+20B9). Bengali block also present (96 codepoints).
- The recognizer is routed by two *fixed* guards, not the network:
- `script.onnx` (hanzi binary classifier, sigmoid scalar; `pipeline.py:274–289`)
sends hanzi lines to `rec_cjk.onnx` β€” irrelevant for Nepali pages
(all-Devanagari pages never trigger it; the classifier is line-level and
trained on hanzi).
- `vocab.canonical` (:42) NFC-normalizes; `Vocab.encode` (:212) drops
out-of-vocab codepoints β€” fine-tuning data must be pre-normalized the
same way or the loss silently degrades (the trainer's data pipeline is
closed; safest is to clone `canonical`/`encode`).
- Confirmed counter-evidence collected in the smoke run: ΰ€Ό (nukta) is often
*dropped contextually* at 32 px height (ΰ€°ΰ€Ό and ΰ€° differ only by a 1–2 px
dot); β‚Ή was read as ΰ€°. Both are training-distribution issues, not capability
gaps β€” exactly the kind of thing fine-tuning would fix (Experiment B).
## 3. Component map: trainable / fixed-but-replaceable / fixed [V]
| Component | Where weights come from | Trainable via shipped artifacts? | Replaceability |
|---|---|---|---|
| `rec.onnx` recognizer | `model.safetensors` (108 tensors, all `rec/`) | **YES** β€” the only shipped state dict; init for a torch CTC retrain (Experiment B) | head/vocab identical β†’ drop-in for +1,678 classes |
| `det.onnx` line detector | ONNX only | NO state dict shipped | could retrain in torch (targets spec exists: `det_post.encode_targets`, `det_post.py:48` β€” `core logit + top/bottom distances` per line, `CORE_FRAC=0.4`, `CORE_X_SHRINK=0.15`) β€” but training code is closed β†’ high effort |
| `order.onnx` reading order/layout | ONNX only | NO | same: `order_post.label_order`/`decode_*` reveal supervision (10 block classes) but no weights |
| `orient.onnx` page rotation | ONNX only | NO | trivial to replace with a torch/tlclassifier or heuristic |
| `mask.onnx` CenterMask | NOT SHIPPED in v3.1.1 (`pipeline.py:201–205` optional) | β€” | absent here; `crop_lines(gray, boxes, 32, None)` runs w/o it (verified) |
| `script.onnx` + `rec_cjk.onnx` | ONNX only | NO | optional; would be excluded from a Devanagari build |
| `math*.onnx` (4 files) | NOT shipped unless `math=True` (`models.py:22–44`) | β€” | out of scope |
| geometry/deskew/layout postproc | pure Python (`pipeline.py:37–149`, `layout_post.py`) | fixed code, no training needed | replaceable with own code paths |
| text-layer (PDF) naive path | `textlayer.py` | fixed code | irrelevant to scans/Nepali notes |
**The decisive asymmetry:** Google ships a *recognizer-only* training starting
point. Fine-tuning anything else means either (a) training from scratch (det
targets spec'd, but no reference weights/recipe) or (b) leaving det/order
frozen ONNX and only exercising the rec head β€” that's Experiment B; both are
documented in `INDRA_EXPERIMENTS.md`.
## 4. Integration architecture (how a page becomes text) [V]
`pipeline.py:456–470` (`OrtOCR.__call__`):
1. `upright()`: optional `orient.onnx` turn above `ORIENT_MIN = 0.95` (:54) +
skew estimate/deskew up to 5Β° (:62–63).
2. `detect()`: `det.onnx` on a page resized long-side to detector input,
threshold β†’ line boxes (uses `det_post.page_to_input` / `decode`;
`ADAPTIVE_MIN_H=9px` re-detect pass at a larger scale, :37–40).
3. `recognize()` (:264–320): CenterMask if present (not shipped); per-line
`crop_line` β†’ `pad_batch` β†’ rec session, CTC collapse & conf gating
(Β§1.2/Β§1.4).
4. line splitting at wide gaps (`SPLIT_GAP=3.0`, `CELL_GAP=1.8`,
`GUTTER_GAP=1.6`, :44–47) and gutter detection (:342).
5. `reading_order()` on `order.onnx` (`feats [1,N,16]`, outputs pairwise salience
`a`, block class `cls` [10 classes], same-block `same`, table t-heads `tab`)
then layout (`layout_post.py: page_blocks`, `page_markdown`).
MIN_CONF=0.5 means **a busy page with low-quality Devanagari can silently
emit empty lines** in Experiment A β€” compare-on-GT-boxes (rec-lines mode)
sidesteps this entirely, which is why Experiment A does both full-pipeline
and rec-lines scoring.
## 5. Licensing [V]
- **Model weights** (`models/indra-ocr-v3.1.1/LICENSE`): **AI PUBS OPEN RAIL-M
(MODIFIED), v0.1, 2023-03-02**. Section II ΒΆ36 states verbatim: *"Use may
include creating any content with, fine-tuning, updating, running, training,
evaluating and/or re-parametrizing the Model"* β€” **fine-tuning is explicitly
licensed**. ΒΆ5 attaches Attachment A use-based restrictions (standard
OpenRAIL misuse list β€” weapons/illegal/defamation; OCR and document
processing is untouched) and requires passing the same restrictions to
distributees; ΒΆ4 permits hosting/redistribution with notice. Derivative
weights (fine-tuned checkpoints, distillations) fall under the same license.
- **Code** (`indra/LICENSE`): **Apache-2.0** β€” reuse, modification,
redistribution of the inference package are standard.
- Practical consequence: our fine-tuned recognizer (Experiment B) is legally a
"Derivative of the Model" under OpenRAIL-M β€” internal deployment and
evaluation are unrestricted; redistribution requires keeping the license +
restriction notice. Full matrix incl. Nepali datasets: `Β§12` of
`INDRA_EXPERIMENTS.md` (dataset side **[B]** β€” needs `docs/DATA_LICENSING.md`
on the remote machine).
## 6. Closing the prior report's [U] items
| Prior unknown | Status | Evidence |
|---|---|---|
| exact recognizer arch (RNN? CNN?) | **[V] + [X]**: pure CNN+CTC, NO RNN/attention | Β§1.1, node dump |
| timestep stride | **[X]**: 4, not 8 | Β§1.3 (graph constants + inc.timings) |
| output contract (best/pmax docstrings vs solo head) | **[V]**: `ids`,`conf`,`lengths` names; truncation mandatory | Β§1.4 |
| safetensors ↔ ONNX mapping (esp. depthwise `onnx::Conv_369…397`) | **[V]**: 108 tensors = 108 initializers, 1,236,366 params both ways; dw weights are `rec/onnx::Conv_369` [128,1,9] + `…370` [128] bias pairs | `inspect_models.py` |
| Devanagari coverage claim | **[V]**: 128/128 codepoints in vocab | `inspect_vocab.py` |
| What re-export must reproduce | **[V]**: mask + per-block re-zeroing + SE-valid-norm + decode tail; `lengths = ceil(widths/4)` clip W/4 | Β§1.2, `inspect_rec_mask.py` |
| Confidence semantics of padded steps | **[X]**: head-on-zeros β‰ˆ blank conf 0.78; caller truncates | Β§1.2(3) |
| vocab[token i+1] ↔ blank 0 | **[V]** | `vocab.py:3,207,241–242` |
| rec input normalization | **[V]**: ink map only (0 paper, 1 ink) per `crop_line`, no mean/std | `rec_preproc.py:1–10` |
| enables fine-tune? | **[V] yes for rec only** (weights exist; no training code; CTC contract reconstructed) | Β§3, `Β§B` doc |
| full pipeline runs on Nepali test image | **[V] smoke**: synthetic shaped page, 10/10 lines, CER 2.6–3.5% | `results/smoke/` |
| what differs in real (non-synth) Nepali | **[B]** frozen remote sets + baseline | Experiment A |
| trained det/order weights for fine-tune | **[V] not shipped** (ONNX only) | Β§3 |
## 7. Open items (not closed by this investigation)
1. **[B]** Zero-shot numbers on the frozen Nepali eval sets (needs `khalti-t4`).
2. **[B]** Head-to-head vs PP-OCRv5 Mobile Devanagari *on the same crops* β€” the
remote machine holds the paddle recognizer + eval harness. The local
smoke CER of 2.6–3.5% is on a synthetic page with a 32 px font and is
*not* a benchmark.
3. **[B]** Licensing matrix final values need `docs/DATA_LICENSING.md` and
`docs/DEVANAGARI_RESOURCES_INVENTORY.md` (remote).
4. FX-detector quality on Devanagari (qubee endpoints, aggressively slanted
conjuncts) is unknowable without the det model being exercised on real
scans β€” that's part of Experiment A full-pipeline.
5. The *root* training repo (referenced by name in the closed code) is not
accessible; Archived recipe unknowns: batch sizes, LR schedule, data mix.
## 8. Reproduce everything
```bash
cd /mnt/c/Users/Dell/Desktop/OCR_INDRA # or wherever the repo is
PY=/tmp/opencode/indra-lib # pip --target deps
export PYTHONPATH="$PWD/indra:$PY"
python3 tools/indra_eval/inspect_vocab.py # vocab + Devanagari coverage
python3 tools/indra_eval/inspect_models.py # graph I/O + param count
python3 tools/indra_eval/inspect_rec_nodes.py # full node dump + smoke run
python3 tools/indra_eval/inspect_rec_mask.py # mask semantics + lengths table
python3 tools/indra_eval/make_synth_page.py results/smoke # synth page + GT
python3 tools/indra_eval/eval_indra.py predict results/smoke/synth_nepali_page.png \
--boxes results/smoke/synth_nepali_gt.jsonl --jsonl results/smoke/rec_only.jsonl
python3 tools/indra_eval/eval_indra.py predict results/smoke/synth_nepali_page.png \
--jsonl results/smoke/full_run.jsonl --print-text
python3 tools/indra_eval/eval_indra.py cer results/smoke/synth_nepali_record.jsonl \
results/smoke/rec_only.jsonl --detail results/smoke/cer_rec_only.json --rec-only
python3 tools/indra_eval/eval_indra.py cer results/smoke/synth_nepali_record.jsonl \
results/smoke/full_run.jsonl --detail results/smoke/cer_full.json
```
Observed (this machine, WSL, CPU, 8 cores, 3 GB RAM): rec-only CER 3.49% /
WER 15.79%; full-pipeline CER 2.62% / WER 13.16%; 10/10 lines found;
mean_conf 0.962–0.963. Timings on this box are meaningless (det β‰ˆ 18.6 min
β€” machine-starved; bench on the T4).