ocr-projects-archive / ocr-tech /docs /LOCAL_RECOGNIZER_SCORING.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
5.19 kB
# Local Devanagari Recognizer Scoring
This is the local CPU path for scoring frozen PaddleOCR line-crop eval packs on
macOS arm64 without a GPU instance. The working route is:
1. export the training checkpoint `.tgz` to PaddleOCR inference format with a
pinned PaddleOCR source checkout, then
2. run `paddleocr._models.text_recognition.TextRecognition(model_dir=...)`, then
3. score with `tools/eval_metrics.py` through `tools/score_recognizer_pack.py`.
Direct `tools/infer_rec.py` checkpoint inference also works on CPU, but the
export route is preferred because it matches the production `TextRecognition`
consumption path used by `spaces/limbu-ocr/app.py`.
## Environment
Use Python 3.11. These commands keep uv and Paddle caches under `/tmp`, which is
important in restricted local/sandboxed runs where `~/.cache` is not writable.
```bash
export UV_CACHE_DIR=/tmp/ocrtech-uv-cache
export OCRTECH_PADDLE_CACHE_ROOT=/tmp/ocrtech-paddle-cache
uv venv --python 3.11 --clear /tmp/ocrtech-paddle311
UV_CACHE_DIR=/tmp/ocrtech-uv-cache uv pip install \
--python /tmp/ocrtech-paddle311/bin/python pip
UV_CACHE_DIR=/tmp/ocrtech-uv-cache uv pip install \
--python /tmp/ocrtech-paddle311/bin/python \
--index-url https://www.paddlepaddle.org.cn/packages/stable/cpu/ \
paddlepaddle==3.3.1
UV_CACHE_DIR=/tmp/ocrtech-uv-cache uv pip install \
--python /tmp/ocrtech-paddle311/bin/python \
--index-url https://pypi.org/simple \
paddleocr==3.7.0 paddlex==3.7.2 rapidfuzz==3.14.5 regex==2026.6.28
git clone --depth 1 --branch v3.7.0 \
https://github.com/PaddlePaddle/PaddleOCR.git \
/tmp/ocrtech-PaddleOCR-v3.7.0
UV_CACHE_DIR=/tmp/ocrtech-uv-cache uv pip install \
--python /tmp/ocrtech-paddle311/bin/python \
--index-url https://pypi.org/simple \
-r /tmp/ocrtech-PaddleOCR-v3.7.0/requirements.txt
export PADDLEOCR_SOURCE=/tmp/ocrtech-PaddleOCR-v3.7.0
```
Verified locally:
- Python `3.11.13`
- `paddlepaddle==3.3.1`
- `paddleocr==3.7.0`
- `paddlex==3.7.2`
- PaddleOCR source tag `v3.7.0`, commit
`b03f46425e8ff4442b268ce449e3eef758146cd4`
- `numpy==2.3.5`, `opencv-python==5.0.0.93`,
`opencv-python-headless==5.0.0.93`,
`opencv-contrib-python==4.10.0.84`, `scikit-image==0.26.0`,
`lmdb==2.2.1`, `albumentations==2.0.8`
## Score A Pack
From the repo root:
```bash
mkdir -p /tmp/ocrtech-deva-v2-artifact
tar -xzf outputs/devanagari-recognizer-v2-2026-06-18/deva-v2.tgz \
-C /tmp/ocrtech-deva-v2-artifact
/tmp/ocrtech-paddle311/bin/python tools/score_recognizer_pack.py \
--model outputs/devanagari-recognizer-v2-2026-06-18/deva-v2.tgz \
--labels path/to/labels.txt \
--dictionary /tmp/ocrtech-deva-v2-artifact/data/dictionaries/devanagari-v2.txt \
--paddleocr-source "$PADDLEOCR_SOURCE" \
--out /tmp/ocrtech-score-run
```
If the model argument is a checkpoint `.tgz`, the scorer exports it to
`/tmp/ocrtech-score-run/exported_model` and then runs `TextRecognition`. If the
model argument is already an exported directory containing `inference.json`,
`inference.pdiparams`, and `inference.yml`, the scorer skips export.
Outputs:
- `predictions.jsonl`: `image`, `gt`, `pred`, `confidence`
- `metrics.json`: route, inputs, and `tools/eval_metrics.py` metrics:
`cer_codepoint`, `cer_grapheme`, `exact_line_acc`
## Smoke Check
The Newar silver labels use relative image paths, so either run against the
original `labels.txt` in place or create an absolute-path 20-line smoke file:
```bash
mkdir -p /tmp/ocrtech-deva-v2-artifact
tar -xzf outputs/devanagari-recognizer-v2-2026-06-18/deva-v2.tgz \
-C /tmp/ocrtech-deva-v2-artifact
base="$PWD/outputs/language-pack-silver-lines-2026-07-04/label-1fefbafa97/"
awk -v base="$base" 'BEGIN { FS="\t"; OFS="\t" } NR <= 20 { print base $1, $2 }' \
outputs/language-pack-silver-lines-2026-07-04/label-1fefbafa97/labels.txt \
> /tmp/ocrtech-newar-smoke20-labels.txt
/tmp/ocrtech-paddle311/bin/python tools/score_recognizer_pack.py \
--model outputs/devanagari-recognizer-v2-2026-06-18/deva-v2.tgz \
--labels /tmp/ocrtech-newar-smoke20-labels.txt \
--dictionary /tmp/ocrtech-deva-v2-artifact/data/dictionaries/devanagari-v2.txt \
--paddleocr-source "$PADDLEOCR_SOURCE" \
--out /tmp/ocrtech-deva-v2-newar-smoke20-score
```
Observed smoke result on the first 20 rows of
`outputs/language-pack-silver-lines-2026-07-04/label-1fefbafa97/labels.txt`:
- codepoint CER: `0.1095679012345679`
- grapheme CER: `0.2512820512820513`
- exact-line accuracy: `0.1`
- `20` lines, `648` codepoints, `390` grapheme clusters
This is only a smoke check, not an official eval number.
## Gotchas
- The checkpoint archive stores a training checkpoint, not an exported inference
model. Export requires PaddleOCR source `tools/export_model.py`.
- The training config's `Global.character_dict_path` is relative to the original
training workspace. Pass the Devanagari dictionary explicitly.
- Under restricted macOS runs, Paddle export may try to write dy2static files to
`~/.cache/paddle`. Set `OCRTECH_PADDLE_CACHE_ROOT=/tmp/ocrtech-paddle-cache`
or run with a writable `HOME`.
- Paddle emits `sysctl` and missing-`ccache` warnings on this machine. They did
not affect export or CPU inference.