File size: 11,688 Bytes
d82db76 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 | # Devanagari recognizer fine-tune (`deva_rec_torch`)
Fine-tunes PaddleOCR's official `devanagari_PP-OCRv5_mobile_rec` for Nepali documents and
serves the result through the unchanged PaddleX runtime. The model that came out of this
pipeline, **devanagari-ft-v3**, has served the Devanagari route of the k2 OCR demo
(`ampixa.thapa-ashish.com.np`) since 2026-09-24. Results and history are in
[`RESEARCH.md`](../../RESEARCH.md) ("Devanagari recognizer fine-tune on k2"); every data
source and its licence is in [`docs/DEVANAGARI_RESOURCES_INVENTORY.md`](../../docs/DEVANAGARI_RESOURCES_INVENTORY.md)
and [`docs/DATA_LICENSING.md`](../../docs/DATA_LICENSING.md).
## Why PyTorch, not Paddle
Paddle training on k2 (Apple M2 Ultra, macOS arm64) runs at about 2 images/s: the PPLCNetV3
depthwise convolutions have no fast CPU kernel there and the wheel has no MKLDNN. The
`transformers` 5.17 PyTorch port of the same network (`PPOCRV5MobileRecForTextRecognition`)
trains at 60-90 images/s on MPS. The pipeline is therefore:
```
PaddlePaddle/devanagari_PP-OCRv5_mobile_rec_safetensors
-> train.py (CTC, MPS) -> HF checkpoint (model.safetensors)
-> export_onnx.py -> inference.onnx + the official inference.yml
-> paddlex.create_model(..., engine="onnxruntime") (same pre/post-processing as the official model)
```
The ONNX graph keeps the official I/O contract: input `x` `[N, 3, 48, W]` float32, output
`fetch_name_0` `[N, T, 570]` probabilities (blank + 568 characters + space).
## Files
| File | Purpose |
|---|---|
| `common.py` | PaddleX-exact preprocessing (`resized_width`, `padded_width`, BGR, `(x/255-0.5)/0.5`), CTC loss/greedy decode, dictionary loading |
| `train.py` | Weighted multi-source CTC fine-tune; width-bucketed batches; frozen BatchNorm statistics; `--guard-dev` no-regression model selection |
| `selection.py` | Guarded checkpoint selection (dependency-free; tested by `tests/test_deva_rec_selection.py`) |
| `export_onnx.py` | HF checkpoint -> PaddleX-servable ONNX directory, verified against torch at widths 320/480/1200 |
| `serve_predict.py` | Predicts eval sets through PaddleX `create_model` (the served path); engine `paddle_static` or `onnxruntime` |
| `score.py` | Scores prediction TSVs with `tools/eval_metrics.py` (micro codepoint + grapheme CER, exact line) |
| `torch_predict.py` | Fast torch-side predictions (equal pad widths per batch) |
| `paddle_predict.py`, `parity.py` | Torch-vs-Paddle parity check of the port |
| `extract_nvidia_en.py` | English line crops from one NVIDIA OCR-Synthetic-Multilingual-v1 HDF5 shard |
| `recipes/cdjk/` | Exact data-build commands run on cdjk (`~/deva-ft-build`, `~/hw-data`), the OFL font selection, the `chunks-v4` build summary |
| `recipes/k2/` | Exact environment setup, training runs v1b -> v2 -> v3 -> v4, eval-set registry and eval runners used on k2 (`~/ocr-deva-ft`), pinned venv requirements |
| `results/` | Scored comparisons behind the published numbers (`final_comparison_v3.json`, `hw_baseline.json`) |
| `page_probe/` | Page-level smoke test on two real government letters: layout probe, crop-strategy probe, hand-written GT, results |
Data builders that live one level up, in `tools/`:
`build_gorkhapatra_devanagari_chunks.py` (real print chunks from Gorkhapatra PDFs),
`render_devanagari_form_synth.py` (synthetic form/office/real-text lines, OFL fonts only),
`relabel_with_kra_fix.py` (corrects labels built before the क्र converter fix),
`compose_dhcd_digit_strings.py` (handwritten digit strings from DHCD).
## Hosts and environments
| Host | Path | Role |
|---|---|---|
| cdjk (Linux) | `~/deva-ft-build/` | Real-chunk and synthetic data builds (CPU) |
| cdjk | `~/hw-data/` | Handwriting downloads and set preparation |
| k2 (Mac Studio, MPS) | `~/ocr-deva-ft/` | Training (`.venv-torch`), served-path evaluation (`.venv-serve`), exports, predictions |
| k2 | `~/ampixa-demo/models/ocr/space/models/` | Deployed models (not a git repo) |
| Khalti T4 (`ampixa@202.51.0.7`, 2x Tesla T4, CUDA) | `~/ocr/` | Training home since 2026-10-01: copy of k2's OCR work plus a git clone of this repo; recipes in `recipes/t4/` |
Do not train or build data on the 18 GB orchestration Mac. k2's GPU is shared with another
project's ASR training: check `ps aux | grep -E "train|asr-node"` before an MPS run, and never
stop another project's job.
On the T4 box, `source recipes/t4/env.sh` (or run the `recipes/t4/*.sh` scripts, which source it):
it sets every path under `~/ocr`, puts `tools/` on `PYTHONPATH` and pins `CUDA_VISIBLE_DEVICES=0`,
because GPU 1 runs another project's job on that shared account. `recipes/t4/setup_env.sh`
builds both venvs inside `~/ocr` (CUDA 12.6 torch 2.14.0; driver 565 supports CUDA <= 12.7);
`recipes/t4/smoke_train.sh` checks CUDA CTC against CPU and runs a short guarded training.
The SSH key for that box (`~/.ssh/khalti-t4`) lives on mbp2 only.
Environments (Python 3.11): `recipes/k2/setup_torch.sh` builds `.venv-torch` and downloads the
official safetensors and ONNX repos; pinned versions are in `recipes/k2/torch-requirements.txt`
(torch 2.14.0, transformers 5.17.0, onnxruntime 1.30.0) and `recipes/k2/serve-requirements.txt`
(paddlepaddle 3.3.1, paddlex 3.7.1, paddleocr 3.7.0, onnxruntime 1.30.0). The scripts import
`eval_metrics` from `tools/`; on k2 the scripts and a copy of `eval_metrics.py` sit together in
`~/ocr-deva-ft/torchft2/`. From a repo checkout, run with `PYTHONPATH=tools`.
## Pipeline
### 1. Data (cdjk)
Label files everywhere use the PaddleOCR format `relative/image/path<TAB>text`.
1. **Real print chunks** (`recipes/cdjk/run_chunks_v4.sh`): 24 Gorkhapatra editions,
split by edition (test 2082-03-24, 2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11),
labels converted from the `nayanepal` text layer with `npttf2utf==0.3.7` plus the
`j|m -> s|` क्र fix. Result `chunks-v4`: 125,017 / 15,570 / 21,821 chunks (build stats in
`recipes/cdjk/chunks-v4-summary.json`). Labels are PDF-text silver, not human-reviewed.
2. **Synthetic lines** (`recipes/cdjk/run_synth*.sh`): citizenship (ne/en), office and
real-text subsets in 44 OFL train families; 4 held-out families (Sura, Khula, Dekko, Sarala)
for test; 8 Latin families (`latin_train`) for English renders; Noto held out.
3. **English replay**: `extract_nvidia_en.py` on `en/validation/validation_000.h5`
(CC BY 4.0; provenance per crop); `recipes/k2/build_v2_lists.py` removes English renders
whose text leaks into the English dev/test lines.
4. **Label hygiene**: `recipes/k2/build_v3_lists.py` drops synthetic lines that carry the old
क्र->व्र converter error; `relabel_with_kra_fix.py` corrects older line labels (E1b).
5. **Handwriting** (`recipes/cdjk/fetch.sh`, `fetch_iiit.sh`, `prep_hw_sets.py`,
`recipes/k2/prep_iiit_k2.py`, `tools/compose_dhcd_digit_strings.py`): H1 Nepali words
(CC0), H2 IIIT-HW-Dev Hindi words (**research/educational use only**), H3 IndicScan (CC BY
4.0, test only), H4 DHCD digit strings (CC BY 4.0; train strings from DHCD Train digits,
test strings from DHCD Test digits).
### 2. Train (k2)
```bash
cd ~/ocr-deva-ft/torchft2
../.venv-torch/bin/python train.py \
--model-dir <init checkpoint> --out ../runs/<run> \
--train "name:root:labels:weight:aug" [...] \
--dev "name:root:labels" [...] \
--guard-dev real_dev --guard-dev synth_dev --guard-dev en_dev --guard-tol 0.002 \
--epochs 3 --batch-size 64 --lr 5e-5 --warmup-steps 300 --eval-every 1500 --max-len 60
```
- `weight` is a per-epoch sampling multiplier of that source's size (0.35 = a random 35% each
epoch, 4.0 = every line four times); `aug` is the probability of capture augmentation.
- BatchNorm running statistics stay frozen: width-bucketed batches have very different
padding fractions, and live statistics made evaluation unstable (v1b step 4000).
- `--guard-dev`: a checkpoint can only become `best` while every guard set's codepoint CER is
within `--guard-tol` of its step-0 value; eligible checkpoints are ranked by the mean CER of
the other dev sets. Use it whenever a run adds a new domain (e.g. handwriting) on top of a
deployed model. Without it the plain mean lets a large gain on the new domain hide a loss
on print.
- Lineage: `run_train_v1b.sh` (from the official safetensors) -> `run_train_v2.sh` ->
`run_train_v3.sh` (on `chunks-v4`, the deployed model) -> `run_train_v4.sh` (handwriting +
about 55% print replay per epoch, guarded; not run yet).
- Launch long runs detached (`nohup ... &`) and poll the log; `train_log.jsonl` in the run
directory records every evaluation.
### 3. Export
```bash
../.venv-torch/bin/python export_onnx.py ../runs/<run>/best \
../hf/devanagari_PP-OCRv5_mobile_rec_onnx ../export/<run>-onnx
```
The export fails unless the ONNX output matches torch (max |diff| < 1e-3) at several widths.
### 4. Evaluate on the served path
```bash
~/ocr-deva-ft/run_serve_eval.sh ~/ocr-deva-ft/export/<run>-onnx <pred_name> # 12 frozen sets from evalsets.sh
```
Every model, including the official one, is scored through PaddleX `create_model` one image
at a time, exactly like the worker's line path, and scored with `tools/eval_metrics.py`. Never
report PaddleOCR's `norm_edit_dis` as CER. `run_hw_baseline.sh` does the same for the
handwriting test sets H1-H3.
A new model must beat the official model on the new domain and stay within 0.2 points of the
deployed model on E1v4, E1b, E7, E3, E4 and E6 before it is deployed.
### 5. Deploy (k2 worker)
Copy the export to `~/ampixa-demo/models/ocr/space/models/<name>/artifacts/` with a
`MODEL-SOURCE.txt`, point `AMPIXA_OCR_DEVANAGARI_DIR` (or the worker default) at it, restart
with `~/ampixa-demo/ops/restart.sh ocr`, and check `lsof -iTCP:8906` and `/health` for a stale
duplicate process. The worker selects `onnxruntime` automatically when `inference.onnx` is
present. Rollback: point the variable at `devanagari-ft-v2/artifacts` or the official
`devanagari-ppocrv5/artifacts`.
## Traps this pipeline already hit
- **Preprocessing parity.** PaddleX pads to `int(48 * max(320/48, w/h))` and caps the resize
width at that value (floor for wide crops). A 1-px difference flipped low-confidence
characters on about 40% of long lines. Batch only crops with equal pad widths: SVTR attention
sees the padding.
- **HF head applies softmax.** Take CTC logits from `model.head.head` (`common.logits`).
- **Official Paddle recipe limits.** `max_text_length` 25, the NRTR branch rejects labels of
23+ characters, and `encode()` silently drops out-of-dictionary characters. `train.py` counts
and reports every skipped label instead.
- **Converter bug.** `npttf2utf` 0.3.7 decodes the `nayanepal` alternate क्र `j|m` as व्र
(and प->फ); 3% of spans. v2 learned it (402 of 530 E7 predictions contained व्र); v3 does
not (2 of 530).
- **Training-time dev numbers are relative only.** `train.py` evaluates dev sets in batches
of 64 padded to the widest crop, not at PaddleX pad widths. v3's selected step logged 8.9%
codepoint CER on `real_dev`, while the served path gives 0.57% on held-out editions. Those
numbers (and the `--guard-dev` comparison) rank checkpoints against step 0; report accuracy
only from `run_serve_eval.sh`. Making the dev loader bucket by pad width is an open fix.
- **Page crops.** The k2 worker crops each detected line with the axis-aligned box of the
detector quad plus 12 px. On a 2°-tilted scan that swallows neighbouring lines: 21.4% CER
versus 2.9% with rotated min-area crops plus 6 px on 43 hand-labelled lines
(`page_probe/`). Line-level eval sets cannot see this; not fixed in the worker yet.
|