ocr-projects-archive / ocr-tech /docs /REMOTE_TRAINING.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
59.4 kB

Remote Training

Target host:

k2@100.97.238.103

Do not store the SSH password in this repository. Use a local secret, ssh-agent, sshpass -e, or a one-off expect wrapper outside version control.

Current Host Check

As of the last verified setup from this workspace:

  • SSH login works.
  • ~/ocr-tech exists and is synced from the local workspace.
  • ~/ocr-tech/.venv/bin/python is 3.11.14.
  • uv is on PATH at ~/.local/bin/uv.
  • Home directory free space is about 352 GiB.
  • nvidia-smi is still not available, so do not claim NVIDIA-backed training there.
  • The machine is macOS 14.6 arm64 on Apple Silicon.

The host is valid for Python 3.11 CPU-side runs and repo automation. It is not an officially supported Paddle training target: the PaddlePaddle installation guide says macOS support requires x86_64, not arm64. Use --require-paddle-training in the audit if you need a host that can honestly be called a PaddleOCR training box.

That restriction is narrower than “no Paddle at all.” As of June 14, 2026, paddlepaddle 3.3.1, paddleocr 3.7.0, and paddlex 3.7.1 do install and run for inference on this Apple Silicon host. The repo can benchmark a stock-Paddle baseline on k2. What is still not honest is calling k2 a supported PaddleOCR training machine.

Audit Command

Use the built-in audit before syncing data or starting training:

export OCRTECH_REMOTE_PASSWORD='...'

ocrtech audit-remote-host \
  --host 100.97.238.103 \
  --user k2 \
  --password-env OCRTECH_REMOTE_PASSWORD \
  --workdir '~/ocr-tech' \
  --require-path '~/ocr-tech' \
  --require-paddle-training \
  --min-python 3.11 \
  --min-free-gb 20 \
  --out outputs/remote-audit

This writes:

outputs/remote-audit/
  remote-audit.json
  remote-audit.md

If the audit fails on Python, disk, platform support, or required commands, fix the host first. Do not push large datasets to a host that already fails the audit.

The audit now checks multiple interpreter candidates such as python3.11, python3.12, python3.13, and ~/.local/bin/python3.11. Do not assume /usr/bin/python3 reflects the usable training interpreter on macOS hosts.

Bootstrap Command

Once the audit passes on Python and disk, provision the remote workdir and virtualenv:

ocrtech bootstrap-remote-host \
  --host 100.97.238.103 \
  --user k2 \
  --password-env OCRTECH_REMOTE_PASSWORD \
  --workdir '~/ocr-tech' \
  --out outputs/remote-bootstrap

This creates:

  • ~/ocr-tech
  • ~/ocr-tech/.venv

It does not sync the repository contents. It only prepares the target path and interpreter environment so the next sync and install step has a valid home.

If the host already has a .venv built with the wrong interpreter, recreate it explicitly:

ocrtech bootstrap-remote-host \
  --host 100.97.238.103 \
  --user k2 \
  --password-env OCRTECH_REMOTE_PASSWORD \
  --workdir '~/ocr-tech' \
  --python-candidate '~/.local/bin/python3.11' \
  --recreate-venv \
  --out outputs/remote-bootstrap

Sync Workspace

Copy the current workspace after bootstrap:

ocrtech sync-remote-workspace \
  --host 100.97.238.103 \
  --user k2 \
  --password-env OCRTECH_REMOTE_PASSWORD \
  --local-root . \
  --remote-workdir '~/ocr-tech' \
  --exclude .venv \
  --exclude outputs \
  --exclude runs \
  --out outputs/remote-sync

This uses a temporary tar archive and remote extract. It is intentionally simple and does not delete remote files that no longer exist locally.

Keep --remote-workdir quoted when it contains ~. If you pass ~/ocr-tech unquoted, your local shell expands it before ocrtech sees the argument, which is the wrong machine.

Expected Setup

On k2, provision a work directory with enough free disk:

mkdir -p ~/ocr-tech
cd ~/ocr-tech
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[datasets,engines]'

If Python 3.11+ is not available, install a newer Python first or use a managed environment. The package targets Python 3.11+.

Sync

From the local workspace:

rsync -az --delete \
  --exclude .venv \
  --exclude .pytest_cache \
  --exclude outputs \
  --exclude runs \
  ./ k2@100.97.238.103:~/ocr-tech/

Never sync a local .venv to a Linux training host. A macOS virtualenv can overwrite the remote console scripts with bad interpreter paths and force a full remote venv repair.

If password auth is required and sshpass is installed locally, prefer SSHPASS over putting the password in command history:

SSHPASS='...' sshpass -e rsync -az ./ k2@100.97.238.103:~/ocr-tech/

Run Training Recipes

Generate manifests and configs locally or remotely:

ocrtech inspect-hf-dataset gauravgiri/nepali-ocr-dataset --split train --limit 3 --out outputs/hf-inspect-training-v1
ocrtech inspect-hf-dataset prashant0919/nepali-synthetic-ocr-lines --split train --limit 3 --out outputs/hf-inspect-training-v1
ocrtech inspect-hf-dataset nvidia/OCR-Synthetic-Multilingual-v1 --hf-subset en --split test --limit 3 --out outputs/hf-inspect-training-v1

ocrtech prepare-data gauravgiri/nepali-ocr-dataset --hf --split train --limit 100 --skip-invalid --out data/manifests
ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines --hf --split train --limit 100 --out data/manifests
ocrtech merge-manifests \
  data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl \
  data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl \
  --out data/manifests/nepali-mixed-all.jsonl
ocrtech audit-manifest data/manifests/nepali-mixed-all.jsonl --out outputs/manifest-audit
ocrtech split-manifest data/manifests/nepali-mixed-all.jsonl --out data/splits
ocrtech train-recognizer \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --out runs/recognizer

For the current mixed Nepali/English line-recognition branch, the verified v2 recipe command on k2 is:

ocrtech train-recognizer \
  --backend paddleocr \
  --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
  --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
  --out runs/paddle-recognizer-mixed-v2 \
  --train-batch-size 16 \
  --eval-batch-size 16 \
  --learning-rate 0.0005 \
  --epochs 10

After generating a PaddleOCR recipe, verify the invariant that each manifest entry maps to exactly one physical label-file line:

uv run python - <<'PY'
from pathlib import Path
from ocrtech.manifest import load_manifest

train_count = len(load_manifest("data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl"))
eval_count = len(load_manifest("data/splits-mixed-v1/eval-with-refs.jsonl"))
train_lines = sum(1 for _ in Path("runs/paddle-recognizer-mixed-v2/labels/train.txt").open())
eval_lines = sum(1 for _ in Path("runs/paddle-recognizer-mixed-v2/labels/eval.txt").open())
assert train_count == train_lines, (train_count, train_lines)
assert eval_count == eval_lines, (eval_count, eval_lines)
PY

The verified v2 counts are 1700/1700 train and 180/180 eval. Earlier recipe generation expanded embedded label newlines and produced too many label lines; do not train from any recipe that fails this invariant.

Recipe generation also verifies dictionary coverage for the exact normalized PaddleOCR labels. If you pass a custom --dictionary, generation fails when any train/eval label contains a character missing from that dictionary. Do not bypass that failure on the remote host; rebuild the dictionary or reject the corpus.

To move the recipe to a supported PaddleOCR training host, create a portable bundle that materializes all referenced images under bundle_assets/images and rewrites manifests/label files to those portable paths:

ocrtech bundle-recognizer-training \
  --recipe-dir runs/paddle-recognizer-mixed-v2 \
  --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
  --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
  --out runs/paddle-recognizer-mixed-v2-bundle \
  --require-corpus-audit \
  --require-eval-real \
  --archive-name paddle-recognizer-mixed-v2-training.tar.gz

Use --require-eval-real for any bundle that may support a paper/admission claim. It fails if the eval manifest has no real slice or contains synthetic rows. Rendered Limbu lines from ocrtech render-synthesis-text-lines belong in train or ablation manifests only; they are not held-out evidence.

Strict audit check on 2026-06-16:

uv run ocrtech audit-recognizer-corpus \
  --train-manifest data/splits-recognizer-next-v1/train.jsonl \
  --eval-manifest data/splits-recognizer-next-v1/eval.jsonl \
  --out outputs/audit-recognizer-next-v1-require-real-eval \
  --min-train-samples 1 \
  --min-eval-samples 1 \
  --min-train-english 0 \
  --min-train-devanagari 0 \
  --min-train-mixed 0 \
  --min-eval-english 0 \
  --min-eval-devanagari 0 \
  --min-eval-mixed 0 \
  --min-train-latin-only 0 \
  --min-train-devanagari-only 0 \
  --min-eval-latin-only 0 \
  --min-eval-devanagari-only 0 \
  --min-train-real 0 \
  --min-train-synthetic 0 \
  --require-eval-real

Result with count thresholds relaxed to isolate the eval-real gate: failed because the eval split still contains synthetic rows (real=15, synthetic=165). This split can support development comparison, but not a paper claim until eval is replaced with held-out real documents.

Verified k2 bundle:

  • archive: runs/paddle-recognizer-mixed-v2-bundle/paddle-recognizer-mixed-v2-training.tar.gz
  • bundle manifest: runs/paddle-recognizer-mixed-v2-bundle/recognizer-training-bundle.json
  • files in archive: 1887
  • materialized images: 1880
  • uncompressed bytes: 79133479
  • compressed archive size: 74061239

After extraction on a supported training host, run PaddleOCR training from the extraction root:

python PaddleOCR/tools/train.py -c runs/paddle-recognizer-mixed-v2/paddle_recognizer.yml

The verified k2 starter run on June 14, 2026 produced:

  • 97 valid gauravgiri samples and 3 explicit rejects
  • 100 valid prashant0919 samples
  • a merged 197-sample corpus
  • a leakage-safe 167 train / 30 eval split
  • runs/recognizer-nepali-starter/paddle_recognizer.yml

The verified k2 schema inspection on June 15, 2026 produced:

  • outputs/hf-inspect-training-v1/gauravgiri__nepali-ocr-dataset-train-inspect.json
  • outputs/hf-inspect-training-v1/prashant0919__nepali-synthetic-ocr-lines-train-inspect.json
  • outputs/hf-inspect-training-v1/nvidia__OCR-Synthetic-Multilingual-v1__en-test-inspect.json

All three passed. Keep this inspection step before conversion so dataset schema drift is caught as a recorded artifact, not discovered halfway through training.

The verified mixed-script HF workflow on the same host later produced:

  • 97 valid gauravgiri Nepali real rows and 3 explicit rejects
  • 100 prashant0919 Nepali synthetic rows
  • 1000 NVIDIA English synthetic line crops from --hf-subset en --split test
  • a merged 1197-sample mixed corpus
  • a leakage-safe 1017 train / 180 eval split
  • materialized eval references at data/references-mixed-v1
  • a rebalanced training manifest with english=850 and nepali=850
  • runs/hf-recognizer-mixed-v1-rebalanced/final-model
  • outputs/experiment-hf-mixed-v1-rebalanced/benchmark/report.json

For k2 specifically, prefer the Hugging Face recognizer backend over Paddle training when you need a real trainable path on Apple Silicon:

python -m pip install -e '.[hf-recognizer]'

ocrtech train-recognizer \
  --backend hf-vision-encoder-decoder \
  --base-model microsoft/trocr-small-printed \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --out runs/hf-recognizer

This does not make k2 a valid official Paddle training host. It gives the repo a separate arm64-compatible recognizer path.

Current active HF-compatible run on k2:

runs/hf-recognizer-mixed-v2-full

Command:

uv run ocrtech train-recognizer \
  --backend hf-vision-encoder-decoder \
  --base-model microsoft/trocr-small-printed \
  --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
  --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
  --out runs/hf-recognizer-mixed-v2-full \
  --train-batch-size 4 \
  --eval-batch-size 4 \
  --learning-rate 0.00005 \
  --epochs 3 \
  --run

Monitor:

PID=$(cat runs/hf-recognizer-mixed-v2-full/train.pid)
ps -p "$PID" -o pid=,stat=,etime=,command=
tail -f runs/hf-recognizer-mixed-v2-full/train.log

Status artifact:

runs/hf-recognizer-mixed-v2-full/train-status.json

Do not treat this run as a model result until runs/hf-recognizer-mixed-v2-full/training_metrics.json and runs/hf-recognizer-mixed-v2-full/final-model/ exist, followed by packaging and benchmarking.

Latest verified checkpoint:

  • k2 time: 2026-06-15 03:35:55 +0545
  • PID: 24642
  • process: running
  • latest observed log progress: 474/1275 trainer steps
  • training_metrics.json: not present
  • final-model/: not present

The run then completed and produced both final-model/ and training_metrics.json.

Training metrics:

  • eval_loss: 2.672821283340454
  • eval_cer: 0.4532251781992946
  • eval_wer: 0.6495539645539645
  • eval_exact_line_accuracy: 0.12222222222222222
  • epoch: 3.0

The package/audit step passed:

  • model card: runs/hf-recognizer-mixed-v2-full/finalized/package/model-card.json
  • audit: runs/hf-recognizer-mixed-v2-full/finalized/audit/model-audit.json
  • audit status: pass

When final-model/ and training_metrics.json exist, finalize the run before benchmarking:

uv run ocrtech finalize-hf-recognizer-run \
  --run-dir runs/hf-recognizer-mixed-v2-full \
  --out runs/hf-recognizer-mixed-v2-full/finalized \
  --model-id ocrtech-hf-mixed-v2-full \
  --base-model microsoft/trocr-small-printed \
  --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
  --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
  --training-summary runs/hf-recognizer-mixed-v2-full/recognizer-training-summary.json \
  --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json

This command intentionally fails if final-model/ or training_metrics.json is missing. If it passes, it writes a packaged model card, model audit, and a post-training report with the next benchmark command.

The first held-out measurement run completed on k2:

  • output: outputs/experiment-hf-recognizer-mixed-v2-full-vs-tesseract
  • baselines: candidate,tesseract
  • rows: 360
  • status: 360 ok, 0 errors
  • candidate CER: 0.4452926496941952
  • Tesseract CER: 0.3290117066116023
  • candidate WER: 0.6551095201095201
  • Tesseract WER: 0.48648528940195623
  • candidate mean latency: 0.4809882701633291s
  • Tesseract mean latency: 0.14839679651065832s
  • validation: skipped because this was a direct branch check, not a claim run

Conclusion: the HF TrOCR branch is a documented negative result. Use k2 for orchestration and evaluation, but spend serious training time next on the PaddleOCR CUDA path.

CUDA PaddleOCR Handoff

The next serious recognizer training step needs a CUDA Linux host. k2 is macOS arm64, so it should not be described as the claim-grade PaddleOCR training machine.

Use the current admitted portable bundle as the handoff artifact:

runs/paddle-recognizer-next-v1-bundle/paddle-recognizer-next-v1-training.tar.gz

After extracting it on the CUDA host and installing/checking out PaddleOCR, run:

python PaddleOCR/tools/train.py -c runs/paddle-recognizer-next-v1/paddle_recognizer.yml

Bring the exported inference model back into this repo, then package, audit, and benchmark it with the same validation flow.

The recognizer-next-v1 bundle was created on k2 after the updated corpus audit passed:

  • train manifest: data/splits-recognizer-next-v1/train.jsonl
  • eval manifest: data/splits-recognizer-next-v1/eval.jsonl
  • train/eval samples: 2212 / 244
  • train Latin/Devanagari/mixed: 1084 / 1174 / 100
  • eval Latin/Devanagari/mixed: 172 / 90 / 28
  • train real/synthetic: 428 / 1272
  • leakage check: pass
  • local mirrored audit: outputs/k2-recognizer-next-v1/recognizer-corpus-audit.json
  • local mirrored bundle manifest: outputs/k2-recognizer-next-v1/recognizer-training-bundle.json

This is only a training handoff. It is not a quality result. A CUDA-trained recognizer still has to clear export, packaging, benchmark, and admission gates before it can replace the current document engine candidate.

Final outcome: recognizer-next-v1 was trained on the restarted AWS G6 L4 host, exported, packaged, benchmarked on hard-50, and rejected. The host was stopped after artifact mirroring.

  • instance: i-0fa19dff7b233a353
  • final state after run: stopped
  • model card: runs/paddle-recognizer-next-v1/package/model-card.json
  • local slim artifact: outputs/aws-g6-paddle-recognizer-next-v1/ocrtech-paddle-recognizer-next-v1-slim-artifacts.tgz
  • slim artifact sha256: 3558cb36fc63e463650d229cd21e1b91955c208cfcd483e419e89d3e86b309ce
  • hard-50 candidate CER/WER: 0.964227 / 1.000000
  • hard-50 Tesseract CER/WER: 0.080249 / 0.258921
  • paired CER wins/losses/ties: 0 / 50 / 0
  • admission decision: reject

Do not relaunch this branch as an engine candidate. The admitted corpus improved the training setup but not recognizer quality.

Follow-up guardrail: ocrtech recognizer-failure-review now aggregates recognizer admission failures across runs and exits nonzero when a model family should be stopped. The current PaddleOCR Devanagari PP-OCRv3 mobile review is:

  • artifact: outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json
  • markdown: outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.md
  • included runs: PaddleOCR mixed v2, v3, v4, and recognizer-next-v1
  • family recommendation: stop_family
  • rejected runs: 4
  • paired failed-gate wins/losses/ties: 0 / 545 / 7

Operational rule: do not start another AWS/G6 job for this Devanagari PP-OCRv3 mobile family unless the recognizer design changes materially. Use GPU time for a different base recognizer, a different objective, or a document-native model that still has to pass the same admission gate.

Enforcement rule for future handoffs: include the failure review when writing and bundling recognizer training jobs:

ocrtech train-recognizer \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --out runs/recognizer \
  --failure-review outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json

ocrtech bundle-recognizer-training \
  --recipe-dir runs/recognizer \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --out runs/recognizer-training-bundle \
  --require-corpus-audit \
  --failure-review outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json

The bundle command reads recognizer-training-summary.json from the actual recipe directory and refuses stopped backend/base-model families before archive creation. That is the guard to run before copying anything to AWS or k2.

The first live TrOCR run on k2 flushed out the actual dependency floor for this backend. Do not hand-install one package at a time. Use the repo extra:

python -m pip install -e '.[hf-recognizer]'

That extra now covers torch, transformers, accelerate, sentencepiece, protobuf, and pillow. On a cold cache, microsoft/trocr-small-printed also downloads about 246 MB of model weights before training starts.

For the verified mixed-script run on k2, use this exact sequence:

ocrtech prepare-data gauravgiri/nepali-ocr-dataset --hf --split train --limit 100 --skip-invalid --out data/manifests
ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines --hf --split train --limit 100 --out data/manifests
ocrtech prepare-data nvidia/OCR-Synthetic-Multilingual-v1 --hf --hf-subset en --split test --limit 1000 --out data/manifests-english-test
ocrtech merge-manifests \
  data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl \
  data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl \
  data/manifests-english-test/nvidia__OCR-Synthetic-Multilingual-v1__en-test.jsonl \
  --out data/manifests/nep-eng-mixed-v1.jsonl
ocrtech audit-manifest data/manifests/nep-eng-mixed-v1.jsonl --out outputs/manifest-audit-mixed-v1 --require-slice english --require-slice nepali
ocrtech split-manifest data/manifests/nep-eng-mixed-v1.jsonl --out data/splits-mixed-v1 --eval-ratio 0.15 --seed 13
ocrtech materialize-references data/splits-mixed-v1/eval.jsonl --out data/references-mixed-v1 --rewritten-manifest data/splits-mixed-v1/eval-with-refs.jsonl
ocrtech rebalance-manifest data/splits-mixed-v1/train.jsonl --out data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl --slice english --slice nepali --seed 13
ocrtech train-recognizer \
  --backend hf-vision-encoder-decoder \
  --base-model microsoft/trocr-small-printed \
  --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
  --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
  --out runs/hf-recognizer-mixed-v1-rebalanced \
  --train-batch-size 4 \
  --eval-batch-size 4 \
  --epochs 0.5 \
  --run
ocrtech package-hf-recognizer \
  --model-dir runs/hf-recognizer-mixed-v1-rebalanced/final-model \
  --out runs/hf-recognizer-mixed-v1-rebalanced/package \
  --model-id ocrtech-hf-mixed-v1-rebalanced \
  --base-model microsoft/trocr-small-printed \
  --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
  --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
  --training-summary runs/hf-recognizer-mixed-v1-rebalanced/recognizer-training-summary.json \
  --metrics-report runs/hf-recognizer-mixed-v1-rebalanced/training_metrics.json \
  --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
ocrtech run-experiment \
  --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
  --out outputs/experiment-hf-mixed-v1-rebalanced \
  --baselines candidate \
  --candidate-model-config runs/hf-recognizer-mixed-v1-rebalanced/package/model-card.json \
  --references-dir data/references-mixed-v1 \
  --skip-validation

Trainer-side eval metrics for that rebalanced run:

  • eval_cer = 0.5983964991975725
  • eval_wer = 0.825085223418557
  • eval_exact_line_accuracy = 0.2611111111111111

Candidate-only benchmark means on the same 180-sample eval split:

  • overall CER: 0.586139
  • English CER: 0.300454
  • Nepali CER: 2.014560
  • real-slice CER: 1.770016

k2 now also has a real tesseract baseline installed via Homebrew, with both eng and nep language data verified. The first direct comparison on that same eval split is still negative for the candidate:

  • candidate overall CER: 0.586139
  • tesseract overall CER: 0.329012
  • candidate Nepali CER: 2.014560
  • tesseract Nepali CER: 0.757175

One shell detail matters on k2: non-interactive SSH sessions do not reliably inherit the Homebrew path. For remote one-shot benchmark commands, export it explicitly before calling ocrtech:

export PATH=/opt/homebrew/bin:$PATH

For the external VLM baselines, the current live state on k2 is now more concrete:

  • paddleocr-vl is runnable through scripts/run_paddleocr_vl.py after installing paddlex[ocr]==3.7.1.
  • glm-ocr does run through scripts/run_glm_ocr.py, but not in the main repo venv. It needed a dedicated Python 3.11 env with git transformers and torchvision.
  • On the first rendered claim-pack blur sample, neither VLM beat raw Tesseract on text CER:
    • glm-ocr: 0.436170
    • paddleocr-vl: 0.521277
    • tesseract: 0.372340

That matters for tesseract baselines and for candidate model cards that instantiate Tesseract internally.

We also verified a small routed Tesseract candidate on k2. A quality_select_composite model card using default Tesseract (psm=6) as primary and psm=13 as secondary, with secondary_quality_margin=0.02, slightly improved the held-out 180-sample mixed eval:

  • raw tesseract overall CER: 0.329012
  • routed tesseract overall CER: 0.324000
  • paired CER versus raw tesseract: 5 wins, 3 losses, 172 ties

That is an engineering improvement, not a frontier claim. The routing margin was tuned on the same pack, and the effect size is small.

We then reran it with proper separation on the same host:

  1. split data/splits-mixed-v1/eval-with-refs.jsonl into a calibration half (89 samples) and untouched holdout half (91 samples)
  2. ran psm=13 vs raw default Tesseract on the calibration half
  3. generated a routed model card with ocrtech calibrate-quality-router
  4. ran that calibrated routed candidate on the untouched holdout half

Holdout result:

  • raw tesseract overall CER: 0.342234
  • calibrated routed tesseract overall CER: 0.340608
  • paired CER on holdout: 2 wins, 1 loss, 88 ties

That is the first out-of-sample confirmation that the routed baseline is not just same-pack leakage. It is still a very small gain.

We also tested Surya on k2 instead of assuming a VLM-style OCR stack would help by default. The host can run it on Apple Silicon after installing:

  • /opt/homebrew/bin/llama-server from llama.cpp
  • surya-ocr inside ~/ocr-tech/.venv

Raw Surya was not a good general baseline on the 91-sample holdout:

  • raw Surya overall CER: 2.198362
  • raw Tesseract overall CER: 0.342234

But Surya was better on the Nepali slice, so we calibrated a routed script_select_composite candidate on the separate 89-sample half with:

python -m ocrtech.cli calibrate-script-router \
  --experiment-report outputs/experiment-surya-calibration-vs-tesseract/experiment.json \
  --out outputs/calibration-surya-script-router \
  --model-id ocrtech-tesseract-surya-script-router \
  --primary-engine tesseract \
  --secondary-engine surya \
  --primary-engine-kwarg language=nep+eng \
  --primary-engine-kwarg psm=6 \
  --secondary-engine-kwarg mode=auto

That selected primary_script_threshold=0.4. On the untouched 91-sample holdout:

  • raw tesseract overall CER: 0.342234
  • calibrated script router overall CER: 0.313084
  • paired CER on holdout: 9 wins, 0 losses, 82 ties

This is the strongest out-of-sample baseline improvement we currently have on k2.

So the host is now baseline-ready, but the model claim is not. The next useful work is not another README victory lap. It is either:

  1. improve the candidate until it actually beats Tesseract on held-out Nepali and mixed-script slices, or
  2. admit this TrOCR path is the wrong recognizer direction and switch to a stronger training backend.

We tested one more recovery path on the same host before stopping: mine Tesseract noisy/clean pairs from the rebalanced training manifest, train a ByT5 corrector, and package it as a text_correction_composite candidate with raw Tesseract as the base engine.

That experiment also failed badly. The corrector trainer finished with:

  • eval_cer = 7.067461343195775
  • eval_wer = 3.334916864181571
  • eval_exact_line_accuracy = 0.0

And the held-out benchmark on the same 180-sample mixed eval was nowhere close to viable:

  • composite overall CER: 6.912065
  • raw Tesseract overall CER: 0.329012
  • paired CER: 0 wins / 170 losses / 10 ties

So k2 remains useful for fast iteration, but the current Tesseract-plus-corrector branch is not a credible direction either.

Then run the generated PaddleOCR command from a checked-out PaddleOCR repo:

python PaddleOCR/tools/train.py -c runs/recognizer/paddle_recognizer.yml

After training, export and package the inference artifacts:

ocrtech export-recognizer \
  --training-config runs/recognizer/paddle_recognizer.yml \
  --checkpoint runs/recognizer/checkpoints/best_accuracy \
  --out runs/recognizer/export \
  --run

ocrtech package-recognizer \
  --inference-dir runs/recognizer/export/inference \
  --dictionary runs/recognizer/nep_eng_dict.txt \
  --out runs/recognizer/package \
  --model-id ocrtech-nep-eng-rec-v1 \
  --train-manifest data/splits/train.jsonl \
  --eval-manifest data/splits/eval.jsonl \
  --training-summary runs/recognizer/recognizer-training-summary.json \
  --admission-validation-report outputs/validation-next-recognizer-hard50/validation.json

ocrtech audit-model runs/recognizer/package/model-card.json

Keep evaluation manifests physically separate from training manifests. Do not use the same image or text hashes across train/eval.

AWS G6 PaddleOCR Run, 2026-06-15

We launched a CUDA host through the playaround AWS profile before the login window expired:

  • instance: i-044a50b7737e8a067
  • type: g6.xlarge
  • GPU: NVIDIA L4, 23034 MiB
  • region: us-east-1
  • AMI: Deep Learning Base OSS Nvidia Driver GPU AMI, Ubuntu 22.04
  • SSH key: ocrtech-g6-training-20260615

The host passed paddle.utils.run_check() with paddlepaddle-gpu==3.3.1. Training used the local PaddleOCR checkout at commit c166448875bcecb8d3b7628fd697ac1c28f8705b.

The original portable recipe from k2 was not valid for actual PaddleOCR training because it was a minimal YAML stub. On AWS we built a corrected config from the upstream Devanagari PP-OCRv3 mobile recognizer config and then patched the local recipe writer to generate that shape directly.

Training command:

cd /opt/ocrtech
source .venv/bin/activate
python PaddleOCR/tools/train.py \
  -c runs/paddle-recognizer-mixed-v2-cuda/paddle_recognizer.yml

Export command:

python PaddleOCR/tools/export_model.py \
  -c runs/paddle-recognizer-mixed-v2-cuda/paddle_recognizer.yml \
  -o Global.pretrained_model=runs/paddle-recognizer-mixed-v2-cuda/checkpoints/best_accuracy \
     Global.save_inference_dir=runs/paddle-recognizer-mixed-v2-cuda/export/inference

Runtime dependencies added for benchmarking:

pip install --no-deps -e /opt/ocrtech/PaddleOCR
pip install paddlex==3.7.1 "paddlex[ocr-core]==3.7.1" aiohttp
sudo apt-get install -y tesseract-ocr tesseract-ocr-eng tesseract-ocr-nep

Packaging caveat for PaddleOCR 3.x:

  • use text_recognition_model_dir, not rec_model_dir
  • set text_recognition_model_name=devanagari_PP-OCRv3_mobile_rec
  • do not pass rec_char_dict_path; exported inference.yml embeds the dictionary

Benchmark command, after materializing references on the AWS host:

cd /opt/ocrtech
source .venv/bin/activate
PYTHONPATH=/opt/ocrtech/ocr-tech/src python -m ocrtech.cli materialize-references \
  data/splits-mixed-v1/eval-with-refs.jsonl \
  --out data/references-mixed-v1 \
  --rewritten-manifest data/splits-mixed-v1/eval-with-refs-aws.jsonl

PYTHONPATH=/opt/ocrtech/ocr-tech/src python -m ocrtech.cli run-experiment \
  --eval-manifest /opt/ocrtech/data/splits-mixed-v1/eval-with-refs.jsonl \
  --out /opt/ocrtech/outputs/experiment-paddle-mixed-v2-cuda-vs-tesseract-v2 \
  --baselines candidate,tesseract \
  --candidate-model-config /opt/ocrtech/runs/paddle-recognizer-mixed-v2-cuda/package/model-card.json \
  --skip-validation

PYTHONPATH=/opt/ocrtech/ocr-tech/src python -m ocrtech.cli rescore-benchmark \
  --benchmark-report outputs/experiment-paddle-mixed-v2-cuda-vs-tesseract-v2/benchmark/report.json \
  --eval-manifest data/splits-mixed-v1/eval-with-refs-aws.jsonl \
  --out outputs/experiment-paddle-mixed-v2-cuda-vs-tesseract-v2/rescored-with-refs

Local archive:

outputs/aws-g6-paddle-mixed-v2-cuda/ocrtech-paddle-mixed-v2-cuda-artifacts.tar.gz

Result:

  • rows: 360, all ok
  • candidate CER/WER: 0.690203 / 0.944234
  • Tesseract CER/WER: 0.364915 / 0.521763
  • candidate latency: 0.077315s
  • Tesseract latency: 0.177510s

This is a real trained model and a real negative result. It proves the CUDA PaddleOCR path works end to end, but it does not support a better-than-Tesseract claim. Before any claim rerun, normalize .img synthetic inputs to supported image suffixes because PaddleOCR logged unsupported-extension warnings for that subset.

Use the manifest image normalizer before the next claim-grade rerun:

ocrtech normalize-manifest-images \
  data/splits-mixed-v1/eval-with-refs.jsonl \
  --out data/splits-mixed-v1/eval-normalized-images

ocrtech materialize-references \
  data/splits-mixed-v1/eval-normalized-images/manifest-images-normalized.jsonl \
  --out data/references-mixed-v1-normalized \
  --rewritten-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl

The normalizer copies already supported image files and converts unsupported file suffixes to PNG with updated image hashes and provenance fields: metadata.source_image_path, metadata.source_image_sha256, and metadata.image_normalization.

On k2, the canonical mixed eval manifest already resolves to .png inputs. After syncing the normalizer and preflight suffix guard, this command produced a clean normalized eval artifact:

ocrtech normalize-manifest-images \
  data/splits-mixed-v1/eval-with-refs.jsonl \
  --out data/splits-mixed-v1/eval-normalized-images

Result:

  • samples: 180
  • copied: 180
  • converted: 0
  • unsupported suffixes: none

Then:

ocrtech materialize-references \
  data/splits-mixed-v1/eval-normalized-images/manifest-images-normalized.jsonl \
  --out data/references-mixed-v1-normalized \
  --rewritten-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl

PATH=/opt/homebrew/bin:$PATH ocrtech preflight-claim \
  --eval-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl \
  --out outputs/preflight-mixed-v1-normalized-script-router-path \
  --baselines candidate,tesseract,surya \
  --candidate-model-config outputs/calibration-surya-script-router/model-card.json \
  --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
  --allow-smoke-eval-pack

Preflight passed on runtime, references, suffixes, candidate audit, component baselines, and calibration separation. It still warns that this is only an OCR line-style eval, not a full document-understanding claim pack.

Local copy:

outputs/k2-mixed-v1-normalized-preflight/k2-mixed-v1-normalized-preflight.tar.gz

2026-06-15 k2 Normalized Mixed Eval Benchmark

Synced the manifest-driven benchmark CLI option to k2 and verified it there:

~/.local/bin/uv run pytest \
  tests/test_cli.py::test_cli_benchmark_loads_inputs_from_manifest -q

Then ran the normalized mixed eval directly from the manifest:

PATH=/opt/homebrew/bin:$PATH ~/.local/bin/uv run ocrtech benchmark \
  --inputs-from-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl \
  --out outputs/live-k2-mixed-v1-normalized-script-router-benchmark \
  --baselines candidate,tesseract,surya \
  --candidate-model-config outputs/calibration-surya-script-router/model-card.json \
  --resume-existing \
  --sample-timeout-seconds 180

The run checkpointed 428 successful rows:

  • candidate: 180/180
  • Tesseract: 180/180
  • Surya: 68/180

The Surya phase stopped progressing after row 68; the benchmark process was terminated manually after preserving report.json. Treat the Surya numbers from this run as partial diagnostics only.

Complete candidate-vs-Tesseract result:

  • candidate CER/WER: 0.354222 / 0.493555
  • Tesseract CER/WER: 0.329012 / 0.486485
  • candidate English CER/WER: 0.272420 / 0.409552
  • Tesseract English CER/WER: 0.243379 / 0.370552
  • candidate Nepali CER/WER: 0.763234 / 0.913571
  • Tesseract Nepali CER/WER: 0.757175 / 1.066151

Local copy:

outputs/k2-mixed-v1-normalized-script-router-benchmark/k2-mixed-v1-normalized-script-router-benchmark.tgz

Interpretation: the normalized line-style rerun is a negative result for the script-router candidate. It is worse than Tesseract overall and on English; its only useful signal is Nepali WER. The next validation-infrastructure fix should be a hard backend watchdog so stalled Surya/VLM calls become row-level errors instead of manual run interruptions.

Follow-up: implemented the hard backend watchdog in ocrtech.benchmark and reran the same normalized benchmark in a fresh output directory:

PATH=/opt/homebrew/bin:$PATH ~/.local/bin/uv run ocrtech benchmark \
  --inputs-from-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl \
  --out outputs/live-k2-mixed-v1-normalized-script-router-benchmark-hard-timeout \
  --baselines candidate,tesseract,surya \
  --candidate-model-config outputs/calibration-surya-script-router/model-card.json \
  --resume-existing \
  --sample-timeout-seconds 45

Completed hard-timeout result:

  • rows: 540
  • candidate: 180/180, all ok
  • Tesseract: 180/180, all ok
  • Surya: 180/180 attempted, 178 ok, 2 timeout errors
  • Surya timeout rows: nvidia__OCR-Synthetic-Multilingual-v1__en-test-000-000013-0000, nvidia__OCR-Synthetic-Multilingual-v1__en-test-000-000028-0002

Summary:

  • candidate CER/WER: 0.354840 / 0.469944
  • Tesseract CER/WER: 0.329012 / 0.486485
  • Surya scored-row CER/WER: 3.157317 / 2.428888

Local copy:

outputs/k2-mixed-v1-normalized-script-router-hard-timeout/k2-mixed-v1-normalized-script-router-hard-timeout.tgz

Interpretation: the watchdog fix works. It converts the previous Surya hang into two row-level timeouts and finishes the full three-system run. The result still does not support the script-router candidate as a better OCR engine.

2026-06-15 k2 PaddleOCR Recognizer Hard-50 Eval Slice

The AWS G6 trained PaddleOCR recognizer lost overall. We ranked its top held-out CER failures and remapped them onto the normalized k2 eval manifest so the next recognizer branch has a runnable hard-eval gate.

Commands on k2:

~/.local/bin/uv run ocrtech export-failure-manifest \
  --source-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl \
  --top-failures outputs/paddle-mixed-v2-cuda-error-analysis/top-failures.json \
  --out data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl \
  --split paddle_hard_eval

~/.local/bin/uv run ocrtech audit-manifest \
  data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl \
  --out outputs/manifest-audit-paddle-mixed-v2-cuda-hard50-normalized \
  --require-slice paddle_hard_eval \
  --require-slice english \
  --require-slice nepali

Result:

  • selected hard failures: 50
  • slice counts: synthetic=50, english=38, en=38, line_crop=38, nepali=12
  • audit: pass

Local copy:

outputs/k2-paddle-mixed-v2-hard50-normalized/k2-paddle-mixed-v2-hard50-normalized.tgz

Interpretation: this hard-50 manifest is held-out evaluation material, not training data. It shows the trained PaddleOCR branch is failing mostly on synthetic English line crops, despite the rebalanced training manifest containing 850 English and 850 Nepali samples. The next recognizer experiment must pass this hard slice while preserving the full normalized mixed eval and document pack gates.

2026-06-15 k2 Line-Level PaddleOCR Composite Rerun

We tested whether the failed AWS G6 PaddleOCR recognizer could still be useful as a selective Devanagari fallback behind a Tesseract primary engine.

The first hard-50 run was invalid because all candidate rows crashed with exit_code=-11. Root cause: hard-timeout mode initialized PaddleOCR in the parent process and then forked child workers. The benchmark now initializes the engine inside each timeout worker. Focused tests passed locally and on k2.

k2 command:

PATH=/opt/homebrew/bin:$PATH ~/.local/bin/uv run ocrtech benchmark \
  --inputs-from-manifest data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl \
  --out outputs/experiment-line-script-paddle-hard50-v2 \
  --baselines candidate,tesseract \
  --candidate-model-config outputs/line-script-paddle-mixed-v2-cuda/model-card.json \
  --resume-existing \
  --sample-timeout-seconds 120

Result:

  • rows: 100
  • candidate: 50/50, all ok
  • Tesseract: 50/50, all ok
  • validation: not_validated
  • hard-50 CER: candidate 0.276489, Tesseract 0.120106
  • English line-crop CER/WER: exact tie with Tesseract, 0.060429 / 0.132259
  • Nepali hard CER/WER: candidate 0.960679 / 1.000000, Tesseract 0.309083 / 0.711607

Local artifacts:

outputs/experiment-line-script-paddle-hard50-v2/report.json
outputs/summary-line-script-paddle-hard50-v2/summary.md
outputs/validation-line-script-paddle-hard50-v2/validation.md

Decision: do not spend AWS G6 time rerunning this exact composite. A CUDA/G6 machine is useful only for the next recognizer training branch: either a better PaddleOCR recipe/base model or a different Devanagari recognizer that first passes the hard-50 admission gate.

AWS G6 PaddleOCR V3 Run, 2026-06-15

We launched a fresh CUDA host while the playaround SSO window was active:

  • instance: i-0b8604ffbcbce25c7
  • type: g6.xlarge
  • GPU: NVIDIA L4, 23034 MiB
  • region: us-east-1
  • public IP at launch: 44.202.252.0
  • AMI: ami-01011b868ec560823, Deep Learning Base OSS Nvidia Driver GPU AMI Ubuntu 22.04, 20260609
  • SSH key: ocrtech-g6-training-20260615
  • security group: sg-05ad6744a62dcabdd, SSH restricted to the current workstation IP used at launch

Runtime setup:

  • repo source snapshot extracted to /opt/ocrtech
  • k2 training bundle copied from runs/paddle-recognizer-mixed-v2-bundle/paddle-recognizer-mixed-v2-training.tar.gz
  • uv installed
  • Python 3.11 venv created
  • installed repo extras: datasets, engines, dev
  • installed paddlepaddle-gpu==3.3.1 from the Paddle CUDA 12.6 index
  • checked out PaddleOCR at c166448875bcecb8d3b7628fd697ac1c28f8705b
  • installed PaddleOCR training requirements
  • paddle.utils.run_check() passed on one GPU

Two repo bugs were fixed before the run could start:

  • integral epoch_num values must be emitted as integers, not floats. Paddle's cosine scheduler rejected 20.0 because T_max must be an int.
  • the PaddleOCR training default must be the official Paddle pretrained .pdparams URL, not the Hugging Face model id. PaddleOCR's training loader could not resolve PaddlePaddle/devanagari_PP-OCRv3_mobile_rec.

V3 training hypothesis:

  • same verified train/eval split as v2: 1700 train, 180 eval
  • lower learning rate: 0.0001
  • longer schedule: 20 epochs
  • official pretrained URL: https://paddle-model-ecology.bj.bcebos.com/paddlex/official_pretrained_model/devanagari_PP-OCRv3_mobile_rec_pretrained.pdparams
  • run directory: /opt/ocrtech/runs/paddle-recognizer-mixed-v3-cuda

Launch command on the G6:

cd /opt/ocrtech
. .venv/bin/activate
ocrtech train-recognizer \
  --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
  --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
  --out runs/paddle-recognizer-mixed-v3-cuda \
  --train-batch-size 16 \
  --eval-batch-size 16 \
  --learning-rate 0.0001 \
  --epochs 20

nohup python PaddleOCR/tools/train.py \
  -c runs/paddle-recognizer-mixed-v3-cuda/paddle_recognizer.yml \
  > runs/paddle-recognizer-mixed-v3-cuda/train.log 2>&1 &

Initial status:

  • process started as PID 4071
  • dataloaders built: 106 train iters, 12 eval iters
  • pretrained weights downloaded and loaded
  • classifier/head weights were shape-mismatched because the expanded dictionary has more classes; PaddleOCR skipped those layers and loaded the rest
  • first logged training step: epoch 1/20, global step 10, loss 137.718468, ETA about 44 minutes

Final training status:

  • completed all 20 epochs
  • best checkpoint: epoch 17
  • best training-reported eval accuracy: 0.09444443919753115
  • best norm edit distance: 0.5833825886055732
  • export succeeded from checkpoints/best_accuracy.pdparams
  • package audit passed for ocrtech-paddle-mixed-v3-cuda

Hard-50 setup correction:

  • the first G6 hard-50 benchmark produced no CER/WER because references had not been materialized on the host.
  • we copied all 50 normalized hard-50 PNGs from k2 and materialized references into data/references-paddle-mixed-v3-hard50.
  • the original hard-50 manifest then audited cleanly on G6.
  • the existing OCR outputs were rescored with eval-paddle-mixed-v2-cuda-hard50-g6-with-refs.jsonl.

Hard-50 result: not_validated.

slice metric candidate Tesseract result
all hard-50 CER 0.921416 0.080249 fail
all hard-50 WER 1.010476 0.258921 fail
English line-crop CER 0.917511 0.031244 fail
English line-crop WER 0.996241 0.114464 fail
Nepali hard CER 0.933782 0.235433 fail
Nepali hard WER 1.055556 0.716369 fail

Local artifact bundle:

outputs/aws-g6-paddle-mixed-v3-cuda/ocrtech-paddle-mixed-v3-cuda-artifacts.tgz

Important interpretation: this v3 run fixed the infrastructure bugs but not the model problem. Lower learning rate plus longer schedule on the same PP-OCRv3 Devanagari recipe still fails the hard-50 admission gate, especially English preservation. Do not run the full mixed eval for this branch.

AWS G6 PaddleOCR V4 Run, 2026-06-15

Current live host:

  • instance: i-0fa19dff7b233a353
  • type: g6.xlarge
  • region/AZ: us-east-1 / us-east-1a
  • public IP: 3.231.208.121
  • AMI: ami-01011b868ec560823
  • SSH key: ocrtech-g6-training-20260615
  • root volume: 200 GiB gp3

Setup already completed on the host:

  • source at /opt/ocrtech
  • mixed-v2 bundle extracted from runs/paddle-recognizer-mixed-v2-bundle/paddle-recognizer-mixed-v2-training.tar.gz
  • Python 3.11.15 venv created with uv
  • paddlepaddle-gpu==3.3.1 installed
  • PaddleOCR checked out at c166448875bcecb8d3b7628fd697ac1c28f8705b
  • PaddleOCR requirements installed
  • paddle.utils.run_check() passed on one NVIDIA L4
  • train manifest audit passed: 1700 samples
  • eval manifest audit passed: 180 samples

V4 hypothesis:

  • same verified mixed-v1 split as v2/v3
  • official Devanagari PP-OCRv3 .pdparams base model
  • lower learning rate than v3: 0.00005
  • 20 epochs
  • batch size 16
  • run directory: /opt/ocrtech/runs/paddle-recognizer-mixed-v4-cuda

Two operational notes matter:

  • The first v4 launch failed before consuming training time because YAML parsed learning_rate: 5e-05 as a string. The recipe generator now emits learning_rate: 0.00005, and local tests cover this.
  • A bad local sync once copied .venv to the host and replaced Linux console scripts with macOS interpreter paths. Repair was done by deleting .venv, recreating it with uv venv --python 3.11, reinstalling repo extras, paddlepaddle-gpu==3.3.1, PaddleOCR requirements, and rerunning paddle.utils.run_check().

When SSH is available, first check whether training is already running:

cd /opt/ocrtech
pgrep -af 'PaddleOCR/tools/train.py|train-v4.sh' || true
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits
tail -80 runs/paddle-recognizer-mixed-v4-cuda/train.log

If it is not running, regenerate the recipe with the fixed local code and launch:

cd /opt/ocrtech
.venv/bin/ocrtech train-recognizer \
  --backend paddleocr \
  --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
  --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
  --out runs/paddle-recognizer-mixed-v4-cuda \
  --train-batch-size 16 \
  --eval-batch-size 16 \
  --learning-rate 0.00005 \
  --epochs 20

grep -E 'learning_rate|epoch_num' runs/paddle-recognizer-mixed-v4-cuda/paddle_recognizer.yml

cat > runs/paddle-recognizer-mixed-v4-cuda/train-v4.sh <<'SH'
#!/usr/bin/env bash
set -euo pipefail
cd /opt/ocrtech
source .venv/bin/activate
export FLAGS_fraction_of_gpu_memory_to_use=0.92
python PaddleOCR/tools/train.py -c runs/paddle-recognizer-mixed-v4-cuda/paddle_recognizer.yml
SH

chmod +x runs/paddle-recognizer-mixed-v4-cuda/train-v4.sh
nohup bash runs/paddle-recognizer-mixed-v4-cuda/train-v4.sh \
  > runs/paddle-recognizer-mixed-v4-cuda/train.log 2>&1 &
echo $! > runs/paddle-recognizer-mixed-v4-cuda/train.pid

After completion, export, package, and include training-summary provenance:

cd /opt/ocrtech
.venv/bin/ocrtech export-recognizer \
  --training-config runs/paddle-recognizer-mixed-v4-cuda/paddle_recognizer.yml \
  --checkpoint runs/paddle-recognizer-mixed-v4-cuda/checkpoints/best_accuracy \
  --out runs/paddle-recognizer-mixed-v4-cuda/export \
  --run

.venv/bin/ocrtech package-recognizer \
  --inference-dir runs/paddle-recognizer-mixed-v4-cuda/export/inference \
  --dictionary runs/paddle-recognizer-mixed-v4-cuda/nep_eng_dict.txt \
  --out runs/paddle-recognizer-mixed-v4-cuda/package \
  --model-id ocrtech-paddle-mixed-v4-cuda \
  --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
  --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
  --training-summary runs/paddle-recognizer-mixed-v4-cuda/recognizer-training-summary.json

.venv/bin/ocrtech audit-model runs/paddle-recognizer-mixed-v4-cuda/package/model-card.json \
  --out runs/paddle-recognizer-mixed-v4-cuda/audit

V4 final status:

  • training completed all 20 epochs on the G6 L4 instance.
  • best PaddleOCR eval metric from training: accuracy 0.016666665740740792, norm edit distance 0.442565389555461, best epoch 19.
  • export succeeded from checkpoint prefix runs/paddle-recognizer-mixed-v4-cuda/checkpoints/best_accuracy.
  • package and model audit succeeded after model-card artifact hashes were fixed to resolve package-relative files.
  • hard-50 admission gate completed with status not_validated.
  • candidate all CER/WER: 0.940500 / 1.028667.
  • Tesseract all CER/WER: 0.080249 / 0.258921.
  • English line-crop CER: candidate 0.950730, Tesseract 0.031244.
  • Nepali CER: candidate 0.908108, Tesseract 0.235433.
  • paired CER wins/losses/ties vs Tesseract: 0 / 50 / 0.
  • local artifact bundle: outputs/aws-g6-paddle-mixed-v4-cuda/ocrtech-paddle-mixed-v4-cuda-artifacts.tgz.
  • stop request completed for instance i-0fa19dff7b233a353; AWS reported final state stopped.

Do not relaunch this exact branch as the engine core. The v4 result is a third negative PaddleOCR mixed-script recognizer result after v2 and v3.

AWS G6 HF Recognizer Next V1

The stopped G6 instance i-0fa19dff7b233a353 was restarted on 2026-06-15 for a new HF recognizer run instead of launching a duplicate GPU host.

Host details at launch:

  • instance type: g6.xlarge
  • public IP: 3.91.25.53
  • SSH key: ~/.ssh/ocrtech-g6-training-20260615.pem
  • remote workspace: /opt/ocrtech
  • GPU: NVIDIA L4, 23034 MiB
  • driver reported by nvidia-smi: 580.159.04
  • torch runtime after install: 2.12.0+cu130
  • CUDA visibility: torch.cuda.is_available() == True

The source files training.py, admission.py, validation.py, and cli.py were synced to /opt/ocrtech/src/ocrtech/. The verified bundle archive was uploaded to:

/opt/ocrtech/incoming/hf-recognizer-next-v1-training.tar.gz

Archive verification:

aff0b277b9cfdd353ed82178c65586353c3511d6db06bd20099cbc62df113873

Dependencies were installed with:

cd /opt/ocrtech
.venv/bin/python -m pip install -e ".[hf-recognizer]"

The bundle was extracted at /opt/ocrtech, then the run was launched with nohup through:

/opt/ocrtech/runs/hf-recognizer-next-v1/train-hf-next-v1.sh

The script runs:

ocrtech train-recognizer \
  --backend hf-vision-encoder-decoder \
  --base-model microsoft/trocr-small-printed \
  --train-manifest data/splits-recognizer-next-v1/train.jsonl \
  --eval-manifest data/splits-recognizer-next-v1/eval.jsonl \
  --out runs/hf-recognizer-next-v1 \
  --max-target-length 128 \
  --train-batch-size 8 \
  --eval-batch-size 8 \
  --learning-rate 5e-05 \
  --epochs 5 \
  --run

Live files:

  • PID file: /opt/ocrtech/runs/hf-recognizer-next-v1/train.pid
  • launch PID: 1862
  • log: /opt/ocrtech/runs/hf-recognizer-next-v1/train.log

Observed shortly after launch:

  • process was alive
  • GPU memory used: about 6500 MiB
  • GPU utilization: 46% to 98%
  • training progress: at least 200 / 1385 trainer steps

Final status:

  • training completed all 5 epochs.
  • final model copied locally to: outputs/aws-g6-hf-recognizer-next-v1/runs-hf-recognizer-next-v1/final-model
  • local package: outputs/aws-g6-hf-recognizer-next-v1/package/model-card.json
  • local audit: outputs/aws-g6-hf-recognizer-next-v1/audit/model-audit.json
  • audit status: passed
  • final eval_cer: 0.18840631911585276
  • final eval_wer: 0.36464600972797695
  • final exact-line accuracy: 0.4344262295081967
  • final eval_loss: 1.0254589319229126

The first attempt to stop the host interrupted a broad rsync before any local artifact copy landed. The same instance was restarted only to retrieve the completed run, then the copy was narrowed to final model files and metadata, excluding trainer checkpoints. The local artifact set intentionally omits checkpoint optimizer state.

After artifact retrieval, the local AWS SSO session had expired and blocked the API stop call. The instance was shut down through SSH instead:

sudo shutdown -h now

The SSH session closed and a later SSH check to the last public IP 3.236.132.219 timed out. Refresh AWS SSO and confirm the EC2 state before any future GPU work:

aws sso login --profile playaround
aws ec2 describe-instances \
  --profile playaround \
  --region us-east-1 \
  --instance-ids i-0fa19dff7b233a353 \
  --query 'Reservations[0].Instances[0].State.Name' \
  --output text

For future runs, do not stop the instance while a PID is still making progress. Stop it as soon as the run completes, fails, or no longer needs the GPU:

aws ec2 stop-instances \
  --profile playaround \
  --region us-east-1 \
  --instance-ids i-0fa19dff7b233a353

2026-06-25 Terminated Instance Recovery Audit

The AWS G6 instance i-0fa19dff7b233a353 is terminated. SSH to the last known public IP times out, and AWS reports:

{"State": "terminated", "PublicIp": null}

Follow-up checks in us-east-1 found no remaining EBS volumes or self-owned snapshots matching the OCR/G6/training filters. That means /opt/ocrtech cannot be inspected anymore. The assessment below is based only on local mirrors and archives already present in this workspace.

Common Recognizer Recovery Bundle

The non-Limbu/common recognizer artifacts were bundled at:

outputs/aws-g6-common-recognizer-recovery-2026-06-25/aws-g6-common-recognizer-important-artifacts-2026-06-25.tgz

Verification:

files: 6008
archive size: 641M
sha256: 12f674c2b8dfde377a359a8af1f33d7a8262e2aa6ccb711e6cd460f9e54a680e
file list: outputs/aws-g6-common-recognizer-recovery-2026-06-25/files.txt

This bundle contains local mirrors for:

  • outputs/aws-g6-paddle-mixed-v2-cuda/
  • outputs/aws-g6-paddle-mixed-v3-cuda/
  • outputs/aws-g6-paddle-mixed-v4-cuda/
  • outputs/aws-g6-paddle-recognizer-next-v1/
  • outputs/aws-g6-hf-recognizer-next-v1/
  • data/splits-recognizer-next-v1/
  • data/splits-mixed-v1/
  • outputs/k2-recognizer-next-v1/
  • outputs/k2-hf-recognizer-next-v1/
  • outputs/k2-hf-recognizer-next-v1-combined-full/
  • validation, admission, error-analysis, and recognizer-failure-review outputs

The local common-recognizer artifacts include generated training configs, training logs, model cards, exported inference packages, HF final model weights, training summaries, benchmark reports, hard-gate validations, and failure-review summaries. The individual Paddle artifact archives contain paddle_recognizer.yml, train.log, exported inference files, package artifacts, and model-card.json; they do not contain a full /opt/ocrtech/src source snapshot.

Local Source Snapshot

Because the remote source tree is gone, the current local dirty source tree was snapshotted at:

outputs/local-source-recovery-2026-06-25/ocr-tech-source-dirty-2026-06-25.tgz

Verification:

files: 421
archive size: 2.1M
sha256: e96e955d1b9ac640802b703e656ab12a7907634673c5bbea8d52f0e41fa1ca8c
file list: outputs/local-source-recovery-2026-06-25/source-files.txt

The local source tree is dirty and should be treated as the best available code state for reproducing recent training/eval tooling. At the time of this audit it had 19 modified tracked source/config/doc/test files and 285 untracked source/config/doc/test files under the archived paths.

What Appears Safe

  • HF recognizer-next-v1: local package and final model weights exist under outputs/aws-g6-hf-recognizer-next-v1/, including model.safetensors, model cards, training config, training log, training metrics, audit, and CPU model card used by later benchmarks.
  • Paddle mixed v2/v3/v4: local AWS artifact archives exist and include packaged inference weights, model cards, configs, logs, and validation/benchmark outputs.
  • Paddle recognizer-next-v1: local slim artifact exists at outputs/aws-g6-paddle-recognizer-next-v1/ocrtech-paddle-recognizer-next-v1-slim-artifacts.tgz with the documented SHA256 3558cb36fc63e463650d229cd21e1b91955c208cfcd483e419e89d3e86b309ce.
  • The common-recognizer train/eval manifests are local: data/splits-recognizer-next-v1/train.jsonl (2212 rows) and data/splits-recognizer-next-v1/eval.jsonl (244 rows).
  • The admission/failure decision is local: PaddleOCR Devanagari PP-OCRv3 mobile common-recognizer family was rejected across mixed v2/v3/v4 and recognizer-next-v1.

What May Be Lost

  • Any source changes made directly on /opt/ocrtech/src, /opt/ocrtech/tools, or the remote PaddleOCR/ checkout after the last local sync and not copied back are unrecoverable.
  • The local AWS artifacts do not include a full remote source-code snapshot, so absence of remote-only code cannot be proven.
  • The Paddle recognizer-next-v1 full checkpoint archive documented at runtime as /tmp/ocrtech-paddle-recognizer-next-v1-artifacts.tgz was not found locally as a full archive. The slim packaged artifact is local, but full checkpoint or optimizer-history bulk from that run should be considered lost unless another copy exists outside this workspace.
  • HF trainer checkpoint optimizer state was intentionally omitted from the local package. The final model is local; intermediate trainer checkpoints are not.
  • Temporary remote files, shell history, notebooks, ad hoc diagnostics, cached downloaded datasets, and any one-off scripts outside the mirrored artifact paths are lost with the terminated instance.

Practical conclusion: model packages, configs, logs, manifests, and result evidence for the common recognizer are mostly preserved locally. The real risk is not model loss; it is remote-only source-code or one-off training/debug scripts. Because the instance was terminated without a source snapshot or EBS snapshot, that risk cannot be closed retroactively.