File size: 11,688 Bytes
d82db76
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
# Devanagari recognizer fine-tune (`deva_rec_torch`)

Fine-tunes PaddleOCR's official `devanagari_PP-OCRv5_mobile_rec` for Nepali documents and
serves the result through the unchanged PaddleX runtime. The model that came out of this
pipeline, **devanagari-ft-v3**, has served the Devanagari route of the k2 OCR demo
(`ampixa.thapa-ashish.com.np`) since 2026-09-24. Results and history are in
[`RESEARCH.md`](../../RESEARCH.md) ("Devanagari recognizer fine-tune on k2"); every data
source and its licence is in [`docs/DEVANAGARI_RESOURCES_INVENTORY.md`](../../docs/DEVANAGARI_RESOURCES_INVENTORY.md)
and [`docs/DATA_LICENSING.md`](../../docs/DATA_LICENSING.md).

## Why PyTorch, not Paddle

Paddle training on k2 (Apple M2 Ultra, macOS arm64) runs at about 2 images/s: the PPLCNetV3
depthwise convolutions have no fast CPU kernel there and the wheel has no MKLDNN. The
`transformers` 5.17 PyTorch port of the same network (`PPOCRV5MobileRecForTextRecognition`)
trains at 60-90 images/s on MPS. The pipeline is therefore:

```
PaddlePaddle/devanagari_PP-OCRv5_mobile_rec_safetensors
  -> train.py (CTC, MPS)  -> HF checkpoint (model.safetensors)
  -> export_onnx.py       -> inference.onnx + the official inference.yml
  -> paddlex.create_model(..., engine="onnxruntime")   (same pre/post-processing as the official model)
```

The ONNX graph keeps the official I/O contract: input `x` `[N, 3, 48, W]` float32, output
`fetch_name_0` `[N, T, 570]` probabilities (blank + 568 characters + space).

## Files

| File | Purpose |
|---|---|
| `common.py` | PaddleX-exact preprocessing (`resized_width`, `padded_width`, BGR, `(x/255-0.5)/0.5`), CTC loss/greedy decode, dictionary loading |
| `train.py` | Weighted multi-source CTC fine-tune; width-bucketed batches; frozen BatchNorm statistics; `--guard-dev` no-regression model selection |
| `selection.py` | Guarded checkpoint selection (dependency-free; tested by `tests/test_deva_rec_selection.py`) |
| `export_onnx.py` | HF checkpoint -> PaddleX-servable ONNX directory, verified against torch at widths 320/480/1200 |
| `serve_predict.py` | Predicts eval sets through PaddleX `create_model` (the served path); engine `paddle_static` or `onnxruntime` |
| `score.py` | Scores prediction TSVs with `tools/eval_metrics.py` (micro codepoint + grapheme CER, exact line) |
| `torch_predict.py` | Fast torch-side predictions (equal pad widths per batch) |
| `paddle_predict.py`, `parity.py` | Torch-vs-Paddle parity check of the port |
| `extract_nvidia_en.py` | English line crops from one NVIDIA OCR-Synthetic-Multilingual-v1 HDF5 shard |
| `recipes/cdjk/` | Exact data-build commands run on cdjk (`~/deva-ft-build`, `~/hw-data`), the OFL font selection, the `chunks-v4` build summary |
| `recipes/k2/` | Exact environment setup, training runs v1b -> v2 -> v3 -> v4, eval-set registry and eval runners used on k2 (`~/ocr-deva-ft`), pinned venv requirements |
| `results/` | Scored comparisons behind the published numbers (`final_comparison_v3.json`, `hw_baseline.json`) |
| `page_probe/` | Page-level smoke test on two real government letters: layout probe, crop-strategy probe, hand-written GT, results |

Data builders that live one level up, in `tools/`:
`build_gorkhapatra_devanagari_chunks.py` (real print chunks from Gorkhapatra PDFs),
`render_devanagari_form_synth.py` (synthetic form/office/real-text lines, OFL fonts only),
`relabel_with_kra_fix.py` (corrects labels built before the क्र converter fix),
`compose_dhcd_digit_strings.py` (handwritten digit strings from DHCD).

## Hosts and environments

| Host | Path | Role |
|---|---|---|
| cdjk (Linux) | `~/deva-ft-build/` | Real-chunk and synthetic data builds (CPU) |
| cdjk | `~/hw-data/` | Handwriting downloads and set preparation |
| k2 (Mac Studio, MPS) | `~/ocr-deva-ft/` | Training (`.venv-torch`), served-path evaluation (`.venv-serve`), exports, predictions |
| k2 | `~/ampixa-demo/models/ocr/space/models/` | Deployed models (not a git repo) |
| Khalti T4 (`ampixa@202.51.0.7`, 2x Tesla T4, CUDA) | `~/ocr/` | Training home since 2026-10-01: copy of k2's OCR work plus a git clone of this repo; recipes in `recipes/t4/` |

Do not train or build data on the 18 GB orchestration Mac. k2's GPU is shared with another
project's ASR training: check `ps aux | grep -E "train|asr-node"` before an MPS run, and never
stop another project's job.

On the T4 box, `source recipes/t4/env.sh` (or run the `recipes/t4/*.sh` scripts, which source it):
it sets every path under `~/ocr`, puts `tools/` on `PYTHONPATH` and pins `CUDA_VISIBLE_DEVICES=0`,
because GPU 1 runs another project's job on that shared account. `recipes/t4/setup_env.sh`
builds both venvs inside `~/ocr` (CUDA 12.6 torch 2.14.0; driver 565 supports CUDA <= 12.7);
`recipes/t4/smoke_train.sh` checks CUDA CTC against CPU and runs a short guarded training.
The SSH key for that box (`~/.ssh/khalti-t4`) lives on mbp2 only.

Environments (Python 3.11): `recipes/k2/setup_torch.sh` builds `.venv-torch` and downloads the
official safetensors and ONNX repos; pinned versions are in `recipes/k2/torch-requirements.txt`
(torch 2.14.0, transformers 5.17.0, onnxruntime 1.30.0) and `recipes/k2/serve-requirements.txt`
(paddlepaddle 3.3.1, paddlex 3.7.1, paddleocr 3.7.0, onnxruntime 1.30.0). The scripts import
`eval_metrics` from `tools/`; on k2 the scripts and a copy of `eval_metrics.py` sit together in
`~/ocr-deva-ft/torchft2/`. From a repo checkout, run with `PYTHONPATH=tools`.

## Pipeline

### 1. Data (cdjk)

Label files everywhere use the PaddleOCR format `relative/image/path<TAB>text`.

1. **Real print chunks** (`recipes/cdjk/run_chunks_v4.sh`): 24 Gorkhapatra editions,
   split by edition (test 2082-03-24, 2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11),
   labels converted from the `nayanepal` text layer with `npttf2utf==0.3.7` plus the
   `j|m -> s|` क्र fix. Result `chunks-v4`: 125,017 / 15,570 / 21,821 chunks (build stats in
   `recipes/cdjk/chunks-v4-summary.json`). Labels are PDF-text silver, not human-reviewed.
2. **Synthetic lines** (`recipes/cdjk/run_synth*.sh`): citizenship (ne/en), office and
   real-text subsets in 44 OFL train families; 4 held-out families (Sura, Khula, Dekko, Sarala)
   for test; 8 Latin families (`latin_train`) for English renders; Noto held out.
3. **English replay**: `extract_nvidia_en.py` on `en/validation/validation_000.h5`
   (CC BY 4.0; provenance per crop); `recipes/k2/build_v2_lists.py` removes English renders
   whose text leaks into the English dev/test lines.
4. **Label hygiene**: `recipes/k2/build_v3_lists.py` drops synthetic lines that carry the old
   क्र->व्र converter error; `relabel_with_kra_fix.py` corrects older line labels (E1b).
5. **Handwriting** (`recipes/cdjk/fetch.sh`, `fetch_iiit.sh`, `prep_hw_sets.py`,
   `recipes/k2/prep_iiit_k2.py`, `tools/compose_dhcd_digit_strings.py`): H1 Nepali words
   (CC0), H2 IIIT-HW-Dev Hindi words (**research/educational use only**), H3 IndicScan (CC BY
   4.0, test only), H4 DHCD digit strings (CC BY 4.0; train strings from DHCD Train digits,
   test strings from DHCD Test digits).

### 2. Train (k2)

```bash
cd ~/ocr-deva-ft/torchft2
../.venv-torch/bin/python train.py \
  --model-dir <init checkpoint> --out ../runs/<run> \
  --train "name:root:labels:weight:aug" [...] \
  --dev "name:root:labels" [...] \
  --guard-dev real_dev --guard-dev synth_dev --guard-dev en_dev --guard-tol 0.002 \
  --epochs 3 --batch-size 64 --lr 5e-5 --warmup-steps 300 --eval-every 1500 --max-len 60
```

- `weight` is a per-epoch sampling multiplier of that source's size (0.35 = a random 35% each
  epoch, 4.0 = every line four times); `aug` is the probability of capture augmentation.
- BatchNorm running statistics stay frozen: width-bucketed batches have very different
  padding fractions, and live statistics made evaluation unstable (v1b step 4000).
- `--guard-dev`: a checkpoint can only become `best` while every guard set's codepoint CER is
  within `--guard-tol` of its step-0 value; eligible checkpoints are ranked by the mean CER of
  the other dev sets. Use it whenever a run adds a new domain (e.g. handwriting) on top of a
  deployed model. Without it the plain mean lets a large gain on the new domain hide a loss
  on print.
- Lineage: `run_train_v1b.sh` (from the official safetensors) -> `run_train_v2.sh` ->
  `run_train_v3.sh` (on `chunks-v4`, the deployed model) -> `run_train_v4.sh` (handwriting +
  about 55% print replay per epoch, guarded; not run yet).
- Launch long runs detached (`nohup ... &`) and poll the log; `train_log.jsonl` in the run
  directory records every evaluation.

### 3. Export

```bash
../.venv-torch/bin/python export_onnx.py ../runs/<run>/best \
  ../hf/devanagari_PP-OCRv5_mobile_rec_onnx ../export/<run>-onnx
```

The export fails unless the ONNX output matches torch (max |diff| < 1e-3) at several widths.

### 4. Evaluate on the served path

```bash
~/ocr-deva-ft/run_serve_eval.sh ~/ocr-deva-ft/export/<run>-onnx <pred_name>   # 12 frozen sets from evalsets.sh
```

Every model, including the official one, is scored through PaddleX `create_model` one image
at a time, exactly like the worker's line path, and scored with `tools/eval_metrics.py`. Never
report PaddleOCR's `norm_edit_dis` as CER. `run_hw_baseline.sh` does the same for the
handwriting test sets H1-H3.

A new model must beat the official model on the new domain and stay within 0.2 points of the
deployed model on E1v4, E1b, E7, E3, E4 and E6 before it is deployed.

### 5. Deploy (k2 worker)

Copy the export to `~/ampixa-demo/models/ocr/space/models/<name>/artifacts/` with a
`MODEL-SOURCE.txt`, point `AMPIXA_OCR_DEVANAGARI_DIR` (or the worker default) at it, restart
with `~/ampixa-demo/ops/restart.sh ocr`, and check `lsof -iTCP:8906` and `/health` for a stale
duplicate process. The worker selects `onnxruntime` automatically when `inference.onnx` is
present. Rollback: point the variable at `devanagari-ft-v2/artifacts` or the official
`devanagari-ppocrv5/artifacts`.

## Traps this pipeline already hit

- **Preprocessing parity.** PaddleX pads to `int(48 * max(320/48, w/h))` and caps the resize
  width at that value (floor for wide crops). A 1-px difference flipped low-confidence
  characters on about 40% of long lines. Batch only crops with equal pad widths: SVTR attention
  sees the padding.
- **HF head applies softmax.** Take CTC logits from `model.head.head` (`common.logits`).
- **Official Paddle recipe limits.** `max_text_length` 25, the NRTR branch rejects labels of
  23+ characters, and `encode()` silently drops out-of-dictionary characters. `train.py` counts
  and reports every skipped label instead.
- **Converter bug.** `npttf2utf` 0.3.7 decodes the `nayanepal` alternate क्र `j|m` as व्र
  (and प->फ); 3% of spans. v2 learned it (402 of 530 E7 predictions contained व्र); v3 does
  not (2 of 530).
- **Training-time dev numbers are relative only.** `train.py` evaluates dev sets in batches
  of 64 padded to the widest crop, not at PaddleX pad widths. v3's selected step logged 8.9%
  codepoint CER on `real_dev`, while the served path gives 0.57% on held-out editions. Those
  numbers (and the `--guard-dev` comparison) rank checkpoints against step 0; report accuracy
  only from `run_serve_eval.sh`. Making the dev loader bucket by pad width is an open fix.
- **Page crops.** The k2 worker crops each detected line with the axis-aligned box of the
  detector quad plus 12 px. On a 2°-tilted scan that swallows neighbouring lines: 21.4% CER
  versus 2.9% with rotated min-area crops plus 6 px on 43 hand-labelled lines
  (`page_probe/`). Line-level eval sets cannot see this; not fixed in the worker yet.