ocr-projects-archive / ocr-tech /docs /LOCAL_RECOGNIZER_SCORING.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
5.19 kB

Local Devanagari Recognizer Scoring

This is the local CPU path for scoring frozen PaddleOCR line-crop eval packs on macOS arm64 without a GPU instance. The working route is:

  1. export the training checkpoint .tgz to PaddleOCR inference format with a pinned PaddleOCR source checkout, then
  2. run paddleocr._models.text_recognition.TextRecognition(model_dir=...), then
  3. score with tools/eval_metrics.py through tools/score_recognizer_pack.py.

Direct tools/infer_rec.py checkpoint inference also works on CPU, but the export route is preferred because it matches the production TextRecognition consumption path used by spaces/limbu-ocr/app.py.

Environment

Use Python 3.11. These commands keep uv and Paddle caches under /tmp, which is important in restricted local/sandboxed runs where ~/.cache is not writable.

export UV_CACHE_DIR=/tmp/ocrtech-uv-cache
export OCRTECH_PADDLE_CACHE_ROOT=/tmp/ocrtech-paddle-cache

uv venv --python 3.11 --clear /tmp/ocrtech-paddle311
UV_CACHE_DIR=/tmp/ocrtech-uv-cache uv pip install \
  --python /tmp/ocrtech-paddle311/bin/python pip
UV_CACHE_DIR=/tmp/ocrtech-uv-cache uv pip install \
  --python /tmp/ocrtech-paddle311/bin/python \
  --index-url https://www.paddlepaddle.org.cn/packages/stable/cpu/ \
  paddlepaddle==3.3.1
UV_CACHE_DIR=/tmp/ocrtech-uv-cache uv pip install \
  --python /tmp/ocrtech-paddle311/bin/python \
  --index-url https://pypi.org/simple \
  paddleocr==3.7.0 paddlex==3.7.2 rapidfuzz==3.14.5 regex==2026.6.28

git clone --depth 1 --branch v3.7.0 \
  https://github.com/PaddlePaddle/PaddleOCR.git \
  /tmp/ocrtech-PaddleOCR-v3.7.0
UV_CACHE_DIR=/tmp/ocrtech-uv-cache uv pip install \
  --python /tmp/ocrtech-paddle311/bin/python \
  --index-url https://pypi.org/simple \
  -r /tmp/ocrtech-PaddleOCR-v3.7.0/requirements.txt

export PADDLEOCR_SOURCE=/tmp/ocrtech-PaddleOCR-v3.7.0

Verified locally:

  • Python 3.11.13
  • paddlepaddle==3.3.1
  • paddleocr==3.7.0
  • paddlex==3.7.2
  • PaddleOCR source tag v3.7.0, commit b03f46425e8ff4442b268ce449e3eef758146cd4
  • numpy==2.3.5, opencv-python==5.0.0.93, opencv-python-headless==5.0.0.93, opencv-contrib-python==4.10.0.84, scikit-image==0.26.0, lmdb==2.2.1, albumentations==2.0.8

Score A Pack

From the repo root:

mkdir -p /tmp/ocrtech-deva-v2-artifact
tar -xzf outputs/devanagari-recognizer-v2-2026-06-18/deva-v2.tgz \
  -C /tmp/ocrtech-deva-v2-artifact

/tmp/ocrtech-paddle311/bin/python tools/score_recognizer_pack.py \
  --model outputs/devanagari-recognizer-v2-2026-06-18/deva-v2.tgz \
  --labels path/to/labels.txt \
  --dictionary /tmp/ocrtech-deva-v2-artifact/data/dictionaries/devanagari-v2.txt \
  --paddleocr-source "$PADDLEOCR_SOURCE" \
  --out /tmp/ocrtech-score-run

If the model argument is a checkpoint .tgz, the scorer exports it to /tmp/ocrtech-score-run/exported_model and then runs TextRecognition. If the model argument is already an exported directory containing inference.json, inference.pdiparams, and inference.yml, the scorer skips export.

Outputs:

  • predictions.jsonl: image, gt, pred, confidence
  • metrics.json: route, inputs, and tools/eval_metrics.py metrics: cer_codepoint, cer_grapheme, exact_line_acc

Smoke Check

The Newar silver labels use relative image paths, so either run against the original labels.txt in place or create an absolute-path 20-line smoke file:

mkdir -p /tmp/ocrtech-deva-v2-artifact
tar -xzf outputs/devanagari-recognizer-v2-2026-06-18/deva-v2.tgz \
  -C /tmp/ocrtech-deva-v2-artifact

base="$PWD/outputs/language-pack-silver-lines-2026-07-04/label-1fefbafa97/"
awk -v base="$base" 'BEGIN { FS="\t"; OFS="\t" } NR <= 20 { print base $1, $2 }' \
  outputs/language-pack-silver-lines-2026-07-04/label-1fefbafa97/labels.txt \
  > /tmp/ocrtech-newar-smoke20-labels.txt

/tmp/ocrtech-paddle311/bin/python tools/score_recognizer_pack.py \
  --model outputs/devanagari-recognizer-v2-2026-06-18/deva-v2.tgz \
  --labels /tmp/ocrtech-newar-smoke20-labels.txt \
  --dictionary /tmp/ocrtech-deva-v2-artifact/data/dictionaries/devanagari-v2.txt \
  --paddleocr-source "$PADDLEOCR_SOURCE" \
  --out /tmp/ocrtech-deva-v2-newar-smoke20-score

Observed smoke result on the first 20 rows of outputs/language-pack-silver-lines-2026-07-04/label-1fefbafa97/labels.txt:

  • codepoint CER: 0.1095679012345679
  • grapheme CER: 0.2512820512820513
  • exact-line accuracy: 0.1
  • 20 lines, 648 codepoints, 390 grapheme clusters

This is only a smoke check, not an official eval number.

Gotchas

  • The checkpoint archive stores a training checkpoint, not an exported inference model. Export requires PaddleOCR source tools/export_model.py.
  • The training config's Global.character_dict_path is relative to the original training workspace. Pass the Devanagari dictionary explicitly.
  • Under restricted macOS runs, Paddle export may try to write dy2static files to ~/.cache/paddle. Set OCRTECH_PADDLE_CACHE_ROOT=/tmp/ocrtech-paddle-cache or run with a writable HOME.
  • Paddle emits sysctl and missing-ccache warnings on this machine. They did not affect export or CPU inference.