Download docs/REMOTE_TRAINING.md from prosperious/ocr-tech-k2: direct link, hf CLI and curl.
- Browser
- Download file 52.9 kB
-
https://huggingface.co/prosperious/ocr-tech-k2/resolve/main/docs/REMOTE_TRAINING.md
- Command line
-
hf download hf://prosperious/ocr-tech-k2/docs/REMOTE_TRAINING.md
-
curl -L -o REMOTE_TRAINING.md https://huggingface.co/prosperious/ocr-tech-k2/resolve/main/docs/REMOTE_TRAINING.md
Remote Training
Target host:
k2@100.97.238.103
Do not store the SSH password in this repository. Use a local secret, ssh-agent, sshpass -e, or a one-off expect wrapper outside version control.
Current Host Check
As of the last verified setup from this workspace:
- SSH login works.
~/ocr-techexists and is synced from the local workspace.~/ocr-tech/.venv/bin/pythonis3.11.14.uvis onPATHat~/.local/bin/uv.- Home directory free space is about
352 GiB. nvidia-smiis still not available, so do not claim NVIDIA-backed training there.- The machine is
macOS 14.6 arm64on Apple Silicon.
The host is valid for Python 3.11 CPU-side runs and repo automation. It is not an officially supported Paddle training target: the PaddlePaddle installation guide says macOS support requires x86_64, not arm64. Use --require-paddle-training in the audit if you need a host that can honestly be called a PaddleOCR training box.
That restriction is narrower than “no Paddle at all.” As of June 14, 2026, paddlepaddle 3.3.1, paddleocr 3.7.0, and paddlex 3.7.1 do install and run for inference on this Apple Silicon host. The repo can benchmark a stock-Paddle baseline on k2. What is still not honest is calling k2 a supported PaddleOCR training machine.
Audit Command
Use the built-in audit before syncing data or starting training:
export OCRTECH_REMOTE_PASSWORD='...'
ocrtech audit-remote-host \
--host 100.97.238.103 \
--user k2 \
--password-env OCRTECH_REMOTE_PASSWORD \
--workdir '~/ocr-tech' \
--require-path '~/ocr-tech' \
--require-paddle-training \
--min-python 3.11 \
--min-free-gb 20 \
--out outputs/remote-audit
This writes:
outputs/remote-audit/
remote-audit.json
remote-audit.md
If the audit fails on Python, disk, platform support, or required commands, fix the host first. Do not push large datasets to a host that already fails the audit.
The audit now checks multiple interpreter candidates such as python3.11, python3.12, python3.13, and ~/.local/bin/python3.11. Do not assume /usr/bin/python3 reflects the usable training interpreter on macOS hosts.
Bootstrap Command
Once the audit passes on Python and disk, provision the remote workdir and virtualenv:
ocrtech bootstrap-remote-host \
--host 100.97.238.103 \
--user k2 \
--password-env OCRTECH_REMOTE_PASSWORD \
--workdir '~/ocr-tech' \
--out outputs/remote-bootstrap
This creates:
~/ocr-tech~/ocr-tech/.venv
It does not sync the repository contents. It only prepares the target path and interpreter environment so the next sync and install step has a valid home.
If the host already has a .venv built with the wrong interpreter, recreate it explicitly:
ocrtech bootstrap-remote-host \
--host 100.97.238.103 \
--user k2 \
--password-env OCRTECH_REMOTE_PASSWORD \
--workdir '~/ocr-tech' \
--python-candidate '~/.local/bin/python3.11' \
--recreate-venv \
--out outputs/remote-bootstrap
Sync Workspace
Copy the current workspace after bootstrap:
ocrtech sync-remote-workspace \
--host 100.97.238.103 \
--user k2 \
--password-env OCRTECH_REMOTE_PASSWORD \
--local-root . \
--remote-workdir '~/ocr-tech' \
--exclude .venv \
--exclude outputs \
--exclude runs \
--out outputs/remote-sync
This uses a temporary tar archive and remote extract. It is intentionally simple and does not delete remote files that no longer exist locally.
Keep --remote-workdir quoted when it contains ~. If you pass ~/ocr-tech unquoted, your local shell expands it before ocrtech sees the argument, which is the wrong machine.
Expected Setup
On k2, provision a work directory with enough free disk:
mkdir -p ~/ocr-tech
cd ~/ocr-tech
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[datasets,engines]'
If Python 3.11+ is not available, install a newer Python first or use a managed environment. The package targets Python 3.11+.
Sync
From the local workspace:
rsync -az --delete \
--exclude .venv \
--exclude .pytest_cache \
--exclude outputs \
--exclude runs \
./ k2@100.97.238.103:~/ocr-tech/
Never sync a local .venv to a Linux training host. A macOS virtualenv can
overwrite the remote console scripts with bad interpreter paths and force a
full remote venv repair.
If password auth is required and sshpass is installed locally, prefer SSHPASS over putting the password in command history:
SSHPASS='...' sshpass -e rsync -az ./ k2@100.97.238.103:~/ocr-tech/
Run Training Recipes
Generate manifests and configs locally or remotely:
ocrtech inspect-hf-dataset gauravgiri/nepali-ocr-dataset --split train --limit 3 --out outputs/hf-inspect-training-v1
ocrtech inspect-hf-dataset prashant0919/nepali-synthetic-ocr-lines --split train --limit 3 --out outputs/hf-inspect-training-v1
ocrtech inspect-hf-dataset nvidia/OCR-Synthetic-Multilingual-v1 --hf-subset en --split test --limit 3 --out outputs/hf-inspect-training-v1
ocrtech prepare-data gauravgiri/nepali-ocr-dataset --hf --split train --limit 100 --skip-invalid --out data/manifests
ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines --hf --split train --limit 100 --out data/manifests
ocrtech merge-manifests \
data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl \
data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl \
--out data/manifests/nepali-mixed-all.jsonl
ocrtech audit-manifest data/manifests/nepali-mixed-all.jsonl --out outputs/manifest-audit
ocrtech split-manifest data/manifests/nepali-mixed-all.jsonl --out data/splits
ocrtech train-recognizer \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--out runs/recognizer
For the current mixed Nepali/English line-recognition branch, the verified v2
recipe command on k2 is:
ocrtech train-recognizer \
--backend paddleocr \
--train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
--eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
--out runs/paddle-recognizer-mixed-v2 \
--train-batch-size 16 \
--eval-batch-size 16 \
--learning-rate 0.0005 \
--epochs 10
After generating a PaddleOCR recipe, verify the invariant that each manifest entry maps to exactly one physical label-file line:
uv run python - <<'PY'
from pathlib import Path
from ocrtech.manifest import load_manifest
train_count = len(load_manifest("data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl"))
eval_count = len(load_manifest("data/splits-mixed-v1/eval-with-refs.jsonl"))
train_lines = sum(1 for _ in Path("runs/paddle-recognizer-mixed-v2/labels/train.txt").open())
eval_lines = sum(1 for _ in Path("runs/paddle-recognizer-mixed-v2/labels/eval.txt").open())
assert train_count == train_lines, (train_count, train_lines)
assert eval_count == eval_lines, (eval_count, eval_lines)
PY
The verified v2 counts are 1700/1700 train and 180/180 eval. Earlier recipe
generation expanded embedded label newlines and produced too many label lines;
do not train from any recipe that fails this invariant.
Recipe generation also verifies dictionary coverage for the exact normalized
PaddleOCR labels. If you pass a custom --dictionary, generation fails when
any train/eval label contains a character missing from that dictionary. Do not
bypass that failure on the remote host; rebuild the dictionary or reject the
corpus.
To move the recipe to a supported PaddleOCR training host, create a portable
bundle that materializes all referenced images under bundle_assets/images and
rewrites manifests/label files to those portable paths:
ocrtech bundle-recognizer-training \
--recipe-dir runs/paddle-recognizer-mixed-v2 \
--train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
--eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
--out runs/paddle-recognizer-mixed-v2-bundle \
--archive-name paddle-recognizer-mixed-v2-training.tar.gz
Verified k2 bundle:
- archive:
runs/paddle-recognizer-mixed-v2-bundle/paddle-recognizer-mixed-v2-training.tar.gz - bundle manifest:
runs/paddle-recognizer-mixed-v2-bundle/recognizer-training-bundle.json - files in archive:
1887 - materialized images:
1880 - uncompressed bytes:
79133479 - compressed archive size:
74061239
After extraction on a supported training host, run PaddleOCR training from the extraction root:
python PaddleOCR/tools/train.py -c runs/paddle-recognizer-mixed-v2/paddle_recognizer.yml
The verified k2 starter run on June 14, 2026 produced:
97validgauravgirisamples and3explicit rejects100validprashant0919samples- a merged
197-sample corpus - a leakage-safe
167train /30eval split runs/recognizer-nepali-starter/paddle_recognizer.yml
The verified k2 schema inspection on June 15, 2026 produced:
outputs/hf-inspect-training-v1/gauravgiri__nepali-ocr-dataset-train-inspect.jsonoutputs/hf-inspect-training-v1/prashant0919__nepali-synthetic-ocr-lines-train-inspect.jsonoutputs/hf-inspect-training-v1/nvidia__OCR-Synthetic-Multilingual-v1__en-test-inspect.json
All three passed. Keep this inspection step before conversion so dataset schema drift is caught as a recorded artifact, not discovered halfway through training.
The verified mixed-script HF workflow on the same host later produced:
97validgauravgiriNepali real rows and3explicit rejects100prashant0919Nepali synthetic rows1000NVIDIA English synthetic line crops from--hf-subset en --split test- a merged
1197-sample mixed corpus - a leakage-safe
1017train /180eval split - materialized eval references at
data/references-mixed-v1 - a rebalanced training manifest with
english=850andnepali=850 runs/hf-recognizer-mixed-v1-rebalanced/final-modeloutputs/experiment-hf-mixed-v1-rebalanced/benchmark/report.json
For k2 specifically, prefer the Hugging Face recognizer backend over Paddle training when you need a real trainable path on Apple Silicon:
python -m pip install -e '.[hf-recognizer]'
ocrtech train-recognizer \
--backend hf-vision-encoder-decoder \
--base-model microsoft/trocr-small-printed \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--out runs/hf-recognizer
This does not make k2 a valid official Paddle training host. It gives the repo a separate arm64-compatible recognizer path.
Current active HF-compatible run on k2:
runs/hf-recognizer-mixed-v2-full
Command:
uv run ocrtech train-recognizer \
--backend hf-vision-encoder-decoder \
--base-model microsoft/trocr-small-printed \
--train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
--eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
--out runs/hf-recognizer-mixed-v2-full \
--train-batch-size 4 \
--eval-batch-size 4 \
--learning-rate 0.00005 \
--epochs 3 \
--run
Monitor:
PID=$(cat runs/hf-recognizer-mixed-v2-full/train.pid)
ps -p "$PID" -o pid=,stat=,etime=,command=
tail -f runs/hf-recognizer-mixed-v2-full/train.log
Status artifact:
runs/hf-recognizer-mixed-v2-full/train-status.json
Do not treat this run as a model result until
runs/hf-recognizer-mixed-v2-full/training_metrics.json and
runs/hf-recognizer-mixed-v2-full/final-model/ exist, followed by packaging and
benchmarking.
Latest verified checkpoint:
- k2 time:
2026-06-15 03:35:55 +0545 - PID:
24642 - process: running
- latest observed log progress:
474/1275trainer steps training_metrics.json: not presentfinal-model/: not present
The run then completed and produced both final-model/ and
training_metrics.json.
Training metrics:
eval_loss:2.672821283340454eval_cer:0.4532251781992946eval_wer:0.6495539645539645eval_exact_line_accuracy:0.12222222222222222epoch:3.0
The package/audit step passed:
- model card:
runs/hf-recognizer-mixed-v2-full/finalized/package/model-card.json - audit:
runs/hf-recognizer-mixed-v2-full/finalized/audit/model-audit.json - audit status: pass
When final-model/ and training_metrics.json exist, finalize the run before
benchmarking:
uv run ocrtech finalize-hf-recognizer-run \
--run-dir runs/hf-recognizer-mixed-v2-full \
--out runs/hf-recognizer-mixed-v2-full/finalized \
--model-id ocrtech-hf-mixed-v2-full \
--base-model microsoft/trocr-small-printed \
--train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
--eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
--training-summary runs/hf-recognizer-mixed-v2-full/recognizer-training-summary.json \
--admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
This command intentionally fails if final-model/ or training_metrics.json
is missing. If it passes, it writes a packaged model card, model audit, and a
post-training report with the next benchmark command.
The first held-out measurement run completed on k2:
- output:
outputs/experiment-hf-recognizer-mixed-v2-full-vs-tesseract - baselines:
candidate,tesseract - rows:
360 - status:
360ok,0errors - candidate CER:
0.4452926496941952 - Tesseract CER:
0.3290117066116023 - candidate WER:
0.6551095201095201 - Tesseract WER:
0.48648528940195623 - candidate mean latency:
0.4809882701633291s - Tesseract mean latency:
0.14839679651065832s - validation: skipped because this was a direct branch check, not a claim run
Conclusion: the HF TrOCR branch is a documented negative result. Use k2 for orchestration and evaluation, but spend serious training time next on the PaddleOCR CUDA path.
CUDA PaddleOCR Handoff
The next serious recognizer training step needs a CUDA Linux host. k2 is macOS arm64, so it should not be described as the claim-grade PaddleOCR training machine.
Use the current admitted portable bundle as the handoff artifact:
runs/paddle-recognizer-next-v1-bundle/paddle-recognizer-next-v1-training.tar.gz
After extracting it on the CUDA host and installing/checking out PaddleOCR, run:
python PaddleOCR/tools/train.py -c runs/paddle-recognizer-next-v1/paddle_recognizer.yml
Bring the exported inference model back into this repo, then package, audit, and benchmark it with the same validation flow.
The recognizer-next-v1 bundle was created on k2 after the updated corpus
audit passed:
- train manifest:
data/splits-recognizer-next-v1/train.jsonl - eval manifest:
data/splits-recognizer-next-v1/eval.jsonl - train/eval samples:
2212/244 - train Latin/Devanagari/mixed:
1084/1174/100 - eval Latin/Devanagari/mixed:
172/90/28 - train real/synthetic:
428/1272 - leakage check: pass
- local mirrored audit:
outputs/k2-recognizer-next-v1/recognizer-corpus-audit.json - local mirrored bundle manifest:
outputs/k2-recognizer-next-v1/recognizer-training-bundle.json
This is only a training handoff. It is not a quality result. A CUDA-trained recognizer still has to clear export, packaging, benchmark, and admission gates before it can replace the current document engine candidate.
Final outcome: recognizer-next-v1 was trained on the restarted AWS G6 L4 host,
exported, packaged, benchmarked on hard-50, and rejected. The host was stopped
after artifact mirroring.
- instance:
i-0fa19dff7b233a353 - final state after run:
stopped - model card:
runs/paddle-recognizer-next-v1/package/model-card.json - local slim artifact:
outputs/aws-g6-paddle-recognizer-next-v1/ocrtech-paddle-recognizer-next-v1-slim-artifacts.tgz - slim artifact sha256:
3558cb36fc63e463650d229cd21e1b91955c208cfcd483e419e89d3e86b309ce - hard-50 candidate CER/WER:
0.964227/1.000000 - hard-50 Tesseract CER/WER:
0.080249/0.258921 - paired CER wins/losses/ties:
0/50/0 - admission decision:
reject
Do not relaunch this branch as an engine candidate. The admitted corpus improved the training setup but not recognizer quality.
Follow-up guardrail: ocrtech recognizer-failure-review now aggregates
recognizer admission failures across runs and exits nonzero when a model family
should be stopped. The current PaddleOCR Devanagari PP-OCRv3 mobile review is:
- artifact:
outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json - markdown:
outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.md - included runs: PaddleOCR mixed v2, v3, v4, and
recognizer-next-v1 - family recommendation:
stop_family - rejected runs:
4 - paired failed-gate wins/losses/ties:
0/545/7
Operational rule: do not start another AWS/G6 job for this Devanagari PP-OCRv3 mobile family unless the recognizer design changes materially. Use GPU time for a different base recognizer, a different objective, or a document-native model that still has to pass the same admission gate.
Enforcement rule for future handoffs: include the failure review when writing and bundling recognizer training jobs:
ocrtech train-recognizer \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--out runs/recognizer \
--failure-review outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json
ocrtech bundle-recognizer-training \
--recipe-dir runs/recognizer \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--out runs/recognizer-training-bundle \
--require-corpus-audit \
--failure-review outputs/recognizer-failure-review-paddle-ppocrv3/recognizer-failure-review.json
The bundle command reads recognizer-training-summary.json from the actual
recipe directory and refuses stopped backend/base-model families before archive
creation. That is the guard to run before copying anything to AWS or k2.
The first live TrOCR run on k2 flushed out the actual dependency floor for this backend. Do not hand-install one package at a time. Use the repo extra:
python -m pip install -e '.[hf-recognizer]'
That extra now covers torch, transformers, accelerate, sentencepiece, protobuf, and pillow. On a cold cache, microsoft/trocr-small-printed also downloads about 246 MB of model weights before training starts.
For the verified mixed-script run on k2, use this exact sequence:
ocrtech prepare-data gauravgiri/nepali-ocr-dataset --hf --split train --limit 100 --skip-invalid --out data/manifests
ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines --hf --split train --limit 100 --out data/manifests
ocrtech prepare-data nvidia/OCR-Synthetic-Multilingual-v1 --hf --hf-subset en --split test --limit 1000 --out data/manifests-english-test
ocrtech merge-manifests \
data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl \
data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl \
data/manifests-english-test/nvidia__OCR-Synthetic-Multilingual-v1__en-test.jsonl \
--out data/manifests/nep-eng-mixed-v1.jsonl
ocrtech audit-manifest data/manifests/nep-eng-mixed-v1.jsonl --out outputs/manifest-audit-mixed-v1 --require-slice english --require-slice nepali
ocrtech split-manifest data/manifests/nep-eng-mixed-v1.jsonl --out data/splits-mixed-v1 --eval-ratio 0.15 --seed 13
ocrtech materialize-references data/splits-mixed-v1/eval.jsonl --out data/references-mixed-v1 --rewritten-manifest data/splits-mixed-v1/eval-with-refs.jsonl
ocrtech rebalance-manifest data/splits-mixed-v1/train.jsonl --out data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl --slice english --slice nepali --seed 13
ocrtech train-recognizer \
--backend hf-vision-encoder-decoder \
--base-model microsoft/trocr-small-printed \
--train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
--eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
--out runs/hf-recognizer-mixed-v1-rebalanced \
--train-batch-size 4 \
--eval-batch-size 4 \
--epochs 0.5 \
--run
ocrtech package-hf-recognizer \
--model-dir runs/hf-recognizer-mixed-v1-rebalanced/final-model \
--out runs/hf-recognizer-mixed-v1-rebalanced/package \
--model-id ocrtech-hf-mixed-v1-rebalanced \
--base-model microsoft/trocr-small-printed \
--train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
--eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
--training-summary runs/hf-recognizer-mixed-v1-rebalanced/recognizer-training-summary.json \
--metrics-report runs/hf-recognizer-mixed-v1-rebalanced/training_metrics.json \
--admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
ocrtech run-experiment \
--eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
--out outputs/experiment-hf-mixed-v1-rebalanced \
--baselines candidate \
--candidate-model-config runs/hf-recognizer-mixed-v1-rebalanced/package/model-card.json \
--references-dir data/references-mixed-v1 \
--skip-validation
Trainer-side eval metrics for that rebalanced run:
eval_cer = 0.5983964991975725eval_wer = 0.825085223418557eval_exact_line_accuracy = 0.2611111111111111
Candidate-only benchmark means on the same 180-sample eval split:
- overall CER:
0.586139 - English CER:
0.300454 - Nepali CER:
2.014560 - real-slice CER:
1.770016
k2 now also has a real tesseract baseline installed via Homebrew, with both eng and nep language data verified. The first direct comparison on that same eval split is still negative for the candidate:
- candidate overall CER:
0.586139 - tesseract overall CER:
0.329012 - candidate Nepali CER:
2.014560 - tesseract Nepali CER:
0.757175
One shell detail matters on k2: non-interactive SSH sessions do not reliably inherit the Homebrew path. For remote one-shot benchmark commands, export it explicitly before calling ocrtech:
export PATH=/opt/homebrew/bin:$PATH
For the external VLM baselines, the current live state on k2 is now more concrete:
paddleocr-vlis runnable throughscripts/run_paddleocr_vl.pyafter installingpaddlex[ocr]==3.7.1.glm-ocrdoes run throughscripts/run_glm_ocr.py, but not in the main repo venv. It needed a dedicated Python 3.11 env with gittransformersandtorchvision.- On the first rendered claim-pack blur sample, neither VLM beat raw Tesseract on text CER:
glm-ocr:0.436170paddleocr-vl:0.521277tesseract:0.372340
That matters for tesseract baselines and for candidate model cards that instantiate Tesseract internally.
We also verified a small routed Tesseract candidate on k2. A quality_select_composite model card using default Tesseract (psm=6) as primary and psm=13 as secondary, with secondary_quality_margin=0.02, slightly improved the held-out 180-sample mixed eval:
- raw tesseract overall CER:
0.329012 - routed tesseract overall CER:
0.324000 - paired CER versus raw tesseract:
5wins,3losses,172ties
That is an engineering improvement, not a frontier claim. The routing margin was tuned on the same pack, and the effect size is small.
We then reran it with proper separation on the same host:
- split
data/splits-mixed-v1/eval-with-refs.jsonlinto a calibration half (89samples) and untouched holdout half (91samples) - ran
psm=13vs raw default Tesseract on the calibration half - generated a routed model card with
ocrtech calibrate-quality-router - ran that calibrated routed candidate on the untouched holdout half
Holdout result:
- raw tesseract overall CER:
0.342234 - calibrated routed tesseract overall CER:
0.340608 - paired CER on holdout:
2wins,1loss,88ties
That is the first out-of-sample confirmation that the routed baseline is not just same-pack leakage. It is still a very small gain.
We also tested Surya on k2 instead of assuming a VLM-style OCR stack would help by default. The host can run it on Apple Silicon after installing:
/opt/homebrew/bin/llama-serverfromllama.cppsurya-ocrinside~/ocr-tech/.venv
Raw Surya was not a good general baseline on the 91-sample holdout:
- raw Surya overall CER:
2.198362 - raw Tesseract overall CER:
0.342234
But Surya was better on the Nepali slice, so we calibrated a routed script_select_composite candidate on the separate 89-sample half with:
python -m ocrtech.cli calibrate-script-router \
--experiment-report outputs/experiment-surya-calibration-vs-tesseract/experiment.json \
--out outputs/calibration-surya-script-router \
--model-id ocrtech-tesseract-surya-script-router \
--primary-engine tesseract \
--secondary-engine surya \
--primary-engine-kwarg language=nep+eng \
--primary-engine-kwarg psm=6 \
--secondary-engine-kwarg mode=auto
That selected primary_script_threshold=0.4. On the untouched 91-sample holdout:
- raw tesseract overall CER:
0.342234 - calibrated script router overall CER:
0.313084 - paired CER on holdout:
9wins,0losses,82ties
This is the strongest out-of-sample baseline improvement we currently have on k2.
So the host is now baseline-ready, but the model claim is not. The next useful work is not another README victory lap. It is either:
- improve the candidate until it actually beats Tesseract on held-out Nepali and mixed-script slices, or
- admit this TrOCR path is the wrong recognizer direction and switch to a stronger training backend.
We tested one more recovery path on the same host before stopping: mine Tesseract noisy/clean pairs from the rebalanced training manifest, train a ByT5 corrector, and package it as a text_correction_composite candidate with raw Tesseract as the base engine.
That experiment also failed badly. The corrector trainer finished with:
eval_cer = 7.067461343195775eval_wer = 3.334916864181571eval_exact_line_accuracy = 0.0
And the held-out benchmark on the same 180-sample mixed eval was nowhere close to viable:
- composite overall CER:
6.912065 - raw Tesseract overall CER:
0.329012 - paired CER:
0wins /170losses /10ties
So k2 remains useful for fast iteration, but the current Tesseract-plus-corrector branch is not a credible direction either.
Then run the generated PaddleOCR command from a checked-out PaddleOCR repo:
python PaddleOCR/tools/train.py -c runs/recognizer/paddle_recognizer.yml
After training, export and package the inference artifacts:
ocrtech export-recognizer \
--training-config runs/recognizer/paddle_recognizer.yml \
--checkpoint runs/recognizer/checkpoints/best_accuracy \
--out runs/recognizer/export \
--run
ocrtech package-recognizer \
--inference-dir runs/recognizer/export/inference \
--dictionary runs/recognizer/nep_eng_dict.txt \
--out runs/recognizer/package \
--model-id ocrtech-nep-eng-rec-v1 \
--train-manifest data/splits/train.jsonl \
--eval-manifest data/splits/eval.jsonl \
--training-summary runs/recognizer/recognizer-training-summary.json \
--admission-validation-report outputs/validation-next-recognizer-hard50/validation.json
ocrtech audit-model runs/recognizer/package/model-card.json
Keep evaluation manifests physically separate from training manifests. Do not use the same image or text hashes across train/eval.
AWS G6 PaddleOCR Run, 2026-06-15
We launched a CUDA host through the playaround AWS profile before the login
window expired:
- instance:
i-044a50b7737e8a067 - type:
g6.xlarge - GPU: NVIDIA L4,
23034 MiB - region:
us-east-1 - AMI: Deep Learning Base OSS Nvidia Driver GPU AMI, Ubuntu 22.04
- SSH key:
ocrtech-g6-training-20260615
The host passed paddle.utils.run_check() with paddlepaddle-gpu==3.3.1.
Training used the local PaddleOCR checkout at commit
c166448875bcecb8d3b7628fd697ac1c28f8705b.
The original portable recipe from k2 was not valid for actual PaddleOCR training because it was a minimal YAML stub. On AWS we built a corrected config from the upstream Devanagari PP-OCRv3 mobile recognizer config and then patched the local recipe writer to generate that shape directly.
Training command:
cd /opt/ocrtech
source .venv/bin/activate
python PaddleOCR/tools/train.py \
-c runs/paddle-recognizer-mixed-v2-cuda/paddle_recognizer.yml
Export command:
python PaddleOCR/tools/export_model.py \
-c runs/paddle-recognizer-mixed-v2-cuda/paddle_recognizer.yml \
-o Global.pretrained_model=runs/paddle-recognizer-mixed-v2-cuda/checkpoints/best_accuracy \
Global.save_inference_dir=runs/paddle-recognizer-mixed-v2-cuda/export/inference
Runtime dependencies added for benchmarking:
pip install --no-deps -e /opt/ocrtech/PaddleOCR
pip install paddlex==3.7.1 "paddlex[ocr-core]==3.7.1" aiohttp
sudo apt-get install -y tesseract-ocr tesseract-ocr-eng tesseract-ocr-nep
Packaging caveat for PaddleOCR 3.x:
- use
text_recognition_model_dir, notrec_model_dir - set
text_recognition_model_name=devanagari_PP-OCRv3_mobile_rec - do not pass
rec_char_dict_path; exportedinference.ymlembeds the dictionary
Benchmark command, after materializing references on the AWS host:
cd /opt/ocrtech
source .venv/bin/activate
PYTHONPATH=/opt/ocrtech/ocr-tech/src python -m ocrtech.cli materialize-references \
data/splits-mixed-v1/eval-with-refs.jsonl \
--out data/references-mixed-v1 \
--rewritten-manifest data/splits-mixed-v1/eval-with-refs-aws.jsonl
PYTHONPATH=/opt/ocrtech/ocr-tech/src python -m ocrtech.cli run-experiment \
--eval-manifest /opt/ocrtech/data/splits-mixed-v1/eval-with-refs.jsonl \
--out /opt/ocrtech/outputs/experiment-paddle-mixed-v2-cuda-vs-tesseract-v2 \
--baselines candidate,tesseract \
--candidate-model-config /opt/ocrtech/runs/paddle-recognizer-mixed-v2-cuda/package/model-card.json \
--skip-validation
PYTHONPATH=/opt/ocrtech/ocr-tech/src python -m ocrtech.cli rescore-benchmark \
--benchmark-report outputs/experiment-paddle-mixed-v2-cuda-vs-tesseract-v2/benchmark/report.json \
--eval-manifest data/splits-mixed-v1/eval-with-refs-aws.jsonl \
--out outputs/experiment-paddle-mixed-v2-cuda-vs-tesseract-v2/rescored-with-refs
Local archive:
outputs/aws-g6-paddle-mixed-v2-cuda/ocrtech-paddle-mixed-v2-cuda-artifacts.tar.gz
Result:
- rows:
360, allok - candidate CER/WER:
0.690203/0.944234 - Tesseract CER/WER:
0.364915/0.521763 - candidate latency:
0.077315s - Tesseract latency:
0.177510s
This is a real trained model and a real negative result. It proves the CUDA
PaddleOCR path works end to end, but it does not support a better-than-Tesseract
claim. Before any claim rerun, normalize .img synthetic inputs to supported
image suffixes because PaddleOCR logged unsupported-extension warnings for that
subset.
Use the manifest image normalizer before the next claim-grade rerun:
ocrtech normalize-manifest-images \
data/splits-mixed-v1/eval-with-refs.jsonl \
--out data/splits-mixed-v1/eval-normalized-images
ocrtech materialize-references \
data/splits-mixed-v1/eval-normalized-images/manifest-images-normalized.jsonl \
--out data/references-mixed-v1-normalized \
--rewritten-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl
The normalizer copies already supported image files and converts unsupported
file suffixes to PNG with updated image hashes and provenance fields:
metadata.source_image_path, metadata.source_image_sha256, and
metadata.image_normalization.
On k2, the canonical mixed eval manifest already resolves to .png inputs.
After syncing the normalizer and preflight suffix guard, this command produced a
clean normalized eval artifact:
ocrtech normalize-manifest-images \
data/splits-mixed-v1/eval-with-refs.jsonl \
--out data/splits-mixed-v1/eval-normalized-images
Result:
- samples:
180 - copied:
180 - converted:
0 - unsupported suffixes: none
Then:
ocrtech materialize-references \
data/splits-mixed-v1/eval-normalized-images/manifest-images-normalized.jsonl \
--out data/references-mixed-v1-normalized \
--rewritten-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl
PATH=/opt/homebrew/bin:$PATH ocrtech preflight-claim \
--eval-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl \
--out outputs/preflight-mixed-v1-normalized-script-router-path \
--baselines candidate,tesseract,surya \
--candidate-model-config outputs/calibration-surya-script-router/model-card.json \
--train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
--allow-smoke-eval-pack
Preflight passed on runtime, references, suffixes, candidate audit, component baselines, and calibration separation. It still warns that this is only an OCR line-style eval, not a full document-understanding claim pack.
Local copy:
outputs/k2-mixed-v1-normalized-preflight/k2-mixed-v1-normalized-preflight.tar.gz
2026-06-15 k2 Normalized Mixed Eval Benchmark
Synced the manifest-driven benchmark CLI option to k2 and verified it there:
~/.local/bin/uv run pytest \
tests/test_cli.py::test_cli_benchmark_loads_inputs_from_manifest -q
Then ran the normalized mixed eval directly from the manifest:
PATH=/opt/homebrew/bin:$PATH ~/.local/bin/uv run ocrtech benchmark \
--inputs-from-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl \
--out outputs/live-k2-mixed-v1-normalized-script-router-benchmark \
--baselines candidate,tesseract,surya \
--candidate-model-config outputs/calibration-surya-script-router/model-card.json \
--resume-existing \
--sample-timeout-seconds 180
The run checkpointed 428 successful rows:
- candidate:
180/180 - Tesseract:
180/180 - Surya:
68/180
The Surya phase stopped progressing after row 68; the benchmark process was
terminated manually after preserving report.json. Treat the Surya numbers from
this run as partial diagnostics only.
Complete candidate-vs-Tesseract result:
- candidate CER/WER:
0.354222/0.493555 - Tesseract CER/WER:
0.329012/0.486485 - candidate English CER/WER:
0.272420/0.409552 - Tesseract English CER/WER:
0.243379/0.370552 - candidate Nepali CER/WER:
0.763234/0.913571 - Tesseract Nepali CER/WER:
0.757175/1.066151
Local copy:
outputs/k2-mixed-v1-normalized-script-router-benchmark/k2-mixed-v1-normalized-script-router-benchmark.tgz
Interpretation: the normalized line-style rerun is a negative result for the script-router candidate. It is worse than Tesseract overall and on English; its only useful signal is Nepali WER. The next validation-infrastructure fix should be a hard backend watchdog so stalled Surya/VLM calls become row-level errors instead of manual run interruptions.
Follow-up: implemented the hard backend watchdog in ocrtech.benchmark and
reran the same normalized benchmark in a fresh output directory:
PATH=/opt/homebrew/bin:$PATH ~/.local/bin/uv run ocrtech benchmark \
--inputs-from-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl \
--out outputs/live-k2-mixed-v1-normalized-script-router-benchmark-hard-timeout \
--baselines candidate,tesseract,surya \
--candidate-model-config outputs/calibration-surya-script-router/model-card.json \
--resume-existing \
--sample-timeout-seconds 45
Completed hard-timeout result:
- rows:
540 - candidate:
180/180, allok - Tesseract:
180/180, allok - Surya:
180/180attempted,178ok,2timeout errors - Surya timeout rows:
nvidia__OCR-Synthetic-Multilingual-v1__en-test-000-000013-0000,nvidia__OCR-Synthetic-Multilingual-v1__en-test-000-000028-0002
Summary:
- candidate CER/WER:
0.354840/0.469944 - Tesseract CER/WER:
0.329012/0.486485 - Surya scored-row CER/WER:
3.157317/2.428888
Local copy:
outputs/k2-mixed-v1-normalized-script-router-hard-timeout/k2-mixed-v1-normalized-script-router-hard-timeout.tgz
Interpretation: the watchdog fix works. It converts the previous Surya hang into two row-level timeouts and finishes the full three-system run. The result still does not support the script-router candidate as a better OCR engine.
2026-06-15 k2 PaddleOCR Recognizer Hard-50 Eval Slice
The AWS G6 trained PaddleOCR recognizer lost overall. We ranked its top held-out CER failures and remapped them onto the normalized k2 eval manifest so the next recognizer branch has a runnable hard-eval gate.
Commands on k2:
~/.local/bin/uv run ocrtech export-failure-manifest \
--source-manifest data/splits-mixed-v1/eval-normalized-images-with-refs.jsonl \
--top-failures outputs/paddle-mixed-v2-cuda-error-analysis/top-failures.json \
--out data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl \
--split paddle_hard_eval
~/.local/bin/uv run ocrtech audit-manifest \
data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl \
--out outputs/manifest-audit-paddle-mixed-v2-cuda-hard50-normalized \
--require-slice paddle_hard_eval \
--require-slice english \
--require-slice nepali
Result:
- selected hard failures:
50 - slice counts:
synthetic=50,english=38,en=38,line_crop=38,nepali=12 - audit: pass
Local copy:
outputs/k2-paddle-mixed-v2-hard50-normalized/k2-paddle-mixed-v2-hard50-normalized.tgz
Interpretation: this hard-50 manifest is held-out evaluation material, not
training data. It shows the trained PaddleOCR branch is failing mostly on
synthetic English line crops, despite the rebalanced training manifest containing
850 English and 850 Nepali samples. The next recognizer experiment must pass
this hard slice while preserving the full normalized mixed eval and document
pack gates.
2026-06-15 k2 Line-Level PaddleOCR Composite Rerun
We tested whether the failed AWS G6 PaddleOCR recognizer could still be useful as a selective Devanagari fallback behind a Tesseract primary engine.
The first hard-50 run was invalid because all candidate rows crashed with
exit_code=-11. Root cause: hard-timeout mode initialized PaddleOCR in the
parent process and then forked child workers. The benchmark now initializes the
engine inside each timeout worker. Focused tests passed locally and on k2.
k2 command:
PATH=/opt/homebrew/bin:$PATH ~/.local/bin/uv run ocrtech benchmark \
--inputs-from-manifest data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl \
--out outputs/experiment-line-script-paddle-hard50-v2 \
--baselines candidate,tesseract \
--candidate-model-config outputs/line-script-paddle-mixed-v2-cuda/model-card.json \
--resume-existing \
--sample-timeout-seconds 120
Result:
- rows:
100 - candidate:
50/50, allok - Tesseract:
50/50, allok - validation:
not_validated - hard-50 CER: candidate
0.276489, Tesseract0.120106 - English line-crop CER/WER: exact tie with Tesseract,
0.060429/0.132259 - Nepali hard CER/WER: candidate
0.960679/1.000000, Tesseract0.309083/0.711607
Local artifacts:
outputs/experiment-line-script-paddle-hard50-v2/report.json
outputs/summary-line-script-paddle-hard50-v2/summary.md
outputs/validation-line-script-paddle-hard50-v2/validation.md
Decision: do not spend AWS G6 time rerunning this exact composite. A CUDA/G6 machine is useful only for the next recognizer training branch: either a better PaddleOCR recipe/base model or a different Devanagari recognizer that first passes the hard-50 admission gate.
AWS G6 PaddleOCR V3 Run, 2026-06-15
We launched a fresh CUDA host while the playaround SSO window was active:
- instance:
i-0b8604ffbcbce25c7 - type:
g6.xlarge - GPU: NVIDIA L4,
23034 MiB - region:
us-east-1 - public IP at launch:
44.202.252.0 - AMI:
ami-01011b868ec560823, Deep Learning Base OSS Nvidia Driver GPU AMI Ubuntu 22.04,20260609 - SSH key:
ocrtech-g6-training-20260615 - security group:
sg-05ad6744a62dcabdd, SSH restricted to the current workstation IP used at launch
Runtime setup:
- repo source snapshot extracted to
/opt/ocrtech - k2 training bundle copied from
runs/paddle-recognizer-mixed-v2-bundle/paddle-recognizer-mixed-v2-training.tar.gz uvinstalled- Python
3.11venv created - installed repo extras:
datasets,engines,dev - installed
paddlepaddle-gpu==3.3.1from the Paddle CUDA 12.6 index - checked out PaddleOCR at
c166448875bcecb8d3b7628fd697ac1c28f8705b - installed PaddleOCR training requirements
paddle.utils.run_check()passed on one GPU
Two repo bugs were fixed before the run could start:
- integral
epoch_numvalues must be emitted as integers, not floats. Paddle's cosine scheduler rejected20.0becauseT_maxmust be anint. - the PaddleOCR training default must be the official Paddle pretrained
.pdparamsURL, not the Hugging Face model id. PaddleOCR's training loader could not resolvePaddlePaddle/devanagari_PP-OCRv3_mobile_rec.
V3 training hypothesis:
- same verified train/eval split as v2:
1700train,180eval - lower learning rate:
0.0001 - longer schedule:
20epochs - official pretrained URL:
https://paddle-model-ecology.bj.bcebos.com/paddlex/official_pretrained_model/devanagari_PP-OCRv3_mobile_rec_pretrained.pdparams - run directory:
/opt/ocrtech/runs/paddle-recognizer-mixed-v3-cuda
Launch command on the G6:
cd /opt/ocrtech
. .venv/bin/activate
ocrtech train-recognizer \
--train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
--eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
--out runs/paddle-recognizer-mixed-v3-cuda \
--train-batch-size 16 \
--eval-batch-size 16 \
--learning-rate 0.0001 \
--epochs 20
nohup python PaddleOCR/tools/train.py \
-c runs/paddle-recognizer-mixed-v3-cuda/paddle_recognizer.yml \
> runs/paddle-recognizer-mixed-v3-cuda/train.log 2>&1 &
Initial status:
- process started as PID
4071 - dataloaders built:
106train iters,12eval iters - pretrained weights downloaded and loaded
- classifier/head weights were shape-mismatched because the expanded dictionary has more classes; PaddleOCR skipped those layers and loaded the rest
- first logged training step: epoch
1/20, global step10, loss137.718468, ETA about44minutes
Final training status:
- completed all
20epochs - best checkpoint: epoch
17 - best training-reported eval accuracy:
0.09444443919753115 - best norm edit distance:
0.5833825886055732 - export succeeded from
checkpoints/best_accuracy.pdparams - package audit passed for
ocrtech-paddle-mixed-v3-cuda
Hard-50 setup correction:
- the first G6 hard-50 benchmark produced no CER/WER because references had not been materialized on the host.
- we copied all
50normalized hard-50 PNGs from k2 and materialized references intodata/references-paddle-mixed-v3-hard50. - the original hard-50 manifest then audited cleanly on G6.
- the existing OCR outputs were rescored with
eval-paddle-mixed-v2-cuda-hard50-g6-with-refs.jsonl.
Hard-50 result: not_validated.
| slice | metric | candidate | Tesseract | result |
|---|---|---|---|---|
| all hard-50 | CER | 0.921416 | 0.080249 | fail |
| all hard-50 | WER | 1.010476 | 0.258921 | fail |
| English line-crop | CER | 0.917511 | 0.031244 | fail |
| English line-crop | WER | 0.996241 | 0.114464 | fail |
| Nepali hard | CER | 0.933782 | 0.235433 | fail |
| Nepali hard | WER | 1.055556 | 0.716369 | fail |
Local artifact bundle:
outputs/aws-g6-paddle-mixed-v3-cuda/ocrtech-paddle-mixed-v3-cuda-artifacts.tgz
Important interpretation: this v3 run fixed the infrastructure bugs but not the model problem. Lower learning rate plus longer schedule on the same PP-OCRv3 Devanagari recipe still fails the hard-50 admission gate, especially English preservation. Do not run the full mixed eval for this branch.
AWS G6 PaddleOCR V4 Run, 2026-06-15
Current live host:
- instance:
i-0fa19dff7b233a353 - type:
g6.xlarge - region/AZ:
us-east-1/us-east-1a - public IP:
3.231.208.121 - AMI:
ami-01011b868ec560823 - SSH key:
ocrtech-g6-training-20260615 - root volume:
200 GiB gp3
Setup already completed on the host:
- source at
/opt/ocrtech - mixed-v2 bundle extracted from
runs/paddle-recognizer-mixed-v2-bundle/paddle-recognizer-mixed-v2-training.tar.gz - Python
3.11.15venv created withuv paddlepaddle-gpu==3.3.1installed- PaddleOCR checked out at
c166448875bcecb8d3b7628fd697ac1c28f8705b - PaddleOCR requirements installed
paddle.utils.run_check()passed on one NVIDIA L4- train manifest audit passed:
1700samples - eval manifest audit passed:
180samples
V4 hypothesis:
- same verified mixed-v1 split as v2/v3
- official Devanagari PP-OCRv3
.pdparamsbase model - lower learning rate than v3:
0.00005 20epochs- batch size
16 - run directory:
/opt/ocrtech/runs/paddle-recognizer-mixed-v4-cuda
Two operational notes matter:
- The first v4 launch failed before consuming training time because YAML parsed
learning_rate: 5e-05as a string. The recipe generator now emitslearning_rate: 0.00005, and local tests cover this. - A bad local sync once copied
.venvto the host and replaced Linux console scripts with macOS interpreter paths. Repair was done by deleting.venv, recreating it withuv venv --python 3.11, reinstalling repo extras,paddlepaddle-gpu==3.3.1, PaddleOCR requirements, and rerunningpaddle.utils.run_check().
When SSH is available, first check whether training is already running:
cd /opt/ocrtech
pgrep -af 'PaddleOCR/tools/train.py|train-v4.sh' || true
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits
tail -80 runs/paddle-recognizer-mixed-v4-cuda/train.log
If it is not running, regenerate the recipe with the fixed local code and launch:
cd /opt/ocrtech
.venv/bin/ocrtech train-recognizer \
--backend paddleocr \
--train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
--eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
--out runs/paddle-recognizer-mixed-v4-cuda \
--train-batch-size 16 \
--eval-batch-size 16 \
--learning-rate 0.00005 \
--epochs 20
grep -E 'learning_rate|epoch_num' runs/paddle-recognizer-mixed-v4-cuda/paddle_recognizer.yml
cat > runs/paddle-recognizer-mixed-v4-cuda/train-v4.sh <<'SH'
#!/usr/bin/env bash
set -euo pipefail
cd /opt/ocrtech
source .venv/bin/activate
export FLAGS_fraction_of_gpu_memory_to_use=0.92
python PaddleOCR/tools/train.py -c runs/paddle-recognizer-mixed-v4-cuda/paddle_recognizer.yml
SH
chmod +x runs/paddle-recognizer-mixed-v4-cuda/train-v4.sh
nohup bash runs/paddle-recognizer-mixed-v4-cuda/train-v4.sh \
> runs/paddle-recognizer-mixed-v4-cuda/train.log 2>&1 &
echo $! > runs/paddle-recognizer-mixed-v4-cuda/train.pid
After completion, export, package, and include training-summary provenance:
cd /opt/ocrtech
.venv/bin/ocrtech export-recognizer \
--training-config runs/paddle-recognizer-mixed-v4-cuda/paddle_recognizer.yml \
--checkpoint runs/paddle-recognizer-mixed-v4-cuda/checkpoints/best_accuracy \
--out runs/paddle-recognizer-mixed-v4-cuda/export \
--run
.venv/bin/ocrtech package-recognizer \
--inference-dir runs/paddle-recognizer-mixed-v4-cuda/export/inference \
--dictionary runs/paddle-recognizer-mixed-v4-cuda/nep_eng_dict.txt \
--out runs/paddle-recognizer-mixed-v4-cuda/package \
--model-id ocrtech-paddle-mixed-v4-cuda \
--train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl \
--eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl \
--training-summary runs/paddle-recognizer-mixed-v4-cuda/recognizer-training-summary.json
.venv/bin/ocrtech audit-model runs/paddle-recognizer-mixed-v4-cuda/package/model-card.json \
--out runs/paddle-recognizer-mixed-v4-cuda/audit
V4 final status:
- training completed all
20epochs on the G6 L4 instance. - best PaddleOCR eval metric from training: accuracy
0.016666665740740792, norm edit distance0.442565389555461, best epoch19. - export succeeded from checkpoint prefix
runs/paddle-recognizer-mixed-v4-cuda/checkpoints/best_accuracy. - package and model audit succeeded after model-card artifact hashes were fixed to resolve package-relative files.
- hard-50 admission gate completed with status
not_validated. - candidate all CER/WER:
0.940500/1.028667. - Tesseract all CER/WER:
0.080249/0.258921. - English line-crop CER: candidate
0.950730, Tesseract0.031244. - Nepali CER: candidate
0.908108, Tesseract0.235433. - paired CER wins/losses/ties vs Tesseract:
0/50/0. - local artifact bundle:
outputs/aws-g6-paddle-mixed-v4-cuda/ocrtech-paddle-mixed-v4-cuda-artifacts.tgz. - stop request completed for instance
i-0fa19dff7b233a353; AWS reported final statestopped.
Do not relaunch this exact branch as the engine core. The v4 result is a third negative PaddleOCR mixed-script recognizer result after v2 and v3.
AWS G6 HF Recognizer Next V1
The stopped G6 instance i-0fa19dff7b233a353 was restarted on 2026-06-15 for a
new HF recognizer run instead of launching a duplicate GPU host.
Host details at launch:
- instance type:
g6.xlarge - public IP:
3.91.25.53 - SSH key:
~/.ssh/ocrtech-g6-training-20260615.pem - remote workspace:
/opt/ocrtech - GPU: NVIDIA L4,
23034MiB - driver reported by
nvidia-smi:580.159.04 - torch runtime after install:
2.12.0+cu130 - CUDA visibility:
torch.cuda.is_available() == True
The source files training.py, admission.py, validation.py, and cli.py
were synced to /opt/ocrtech/src/ocrtech/. The verified bundle archive was
uploaded to:
/opt/ocrtech/incoming/hf-recognizer-next-v1-training.tar.gz
Archive verification:
aff0b277b9cfdd353ed82178c65586353c3511d6db06bd20099cbc62df113873
Dependencies were installed with:
cd /opt/ocrtech
.venv/bin/python -m pip install -e ".[hf-recognizer]"
The bundle was extracted at /opt/ocrtech, then the run was launched with
nohup through:
/opt/ocrtech/runs/hf-recognizer-next-v1/train-hf-next-v1.sh
The script runs:
ocrtech train-recognizer \
--backend hf-vision-encoder-decoder \
--base-model microsoft/trocr-small-printed \
--train-manifest data/splits-recognizer-next-v1/train.jsonl \
--eval-manifest data/splits-recognizer-next-v1/eval.jsonl \
--out runs/hf-recognizer-next-v1 \
--max-target-length 128 \
--train-batch-size 8 \
--eval-batch-size 8 \
--learning-rate 5e-05 \
--epochs 5 \
--run
Live files:
- PID file:
/opt/ocrtech/runs/hf-recognizer-next-v1/train.pid - launch PID:
1862 - log:
/opt/ocrtech/runs/hf-recognizer-next-v1/train.log
Observed shortly after launch:
- process was alive
- GPU memory used: about
6500MiB - GPU utilization:
46%to98% - training progress: at least
200/1385trainer steps
Final status:
- training completed all
5epochs. - final model copied locally to:
outputs/aws-g6-hf-recognizer-next-v1/runs-hf-recognizer-next-v1/final-model - local package:
outputs/aws-g6-hf-recognizer-next-v1/package/model-card.json - local audit:
outputs/aws-g6-hf-recognizer-next-v1/audit/model-audit.json - audit status: passed
- final
eval_cer:0.18840631911585276 - final
eval_wer:0.36464600972797695 - final exact-line accuracy:
0.4344262295081967 - final
eval_loss:1.0254589319229126
The first attempt to stop the host interrupted a broad rsync before any local artifact copy landed. The same instance was restarted only to retrieve the completed run, then the copy was narrowed to final model files and metadata, excluding trainer checkpoints. The local artifact set intentionally omits checkpoint optimizer state.
After artifact retrieval, the local AWS SSO session had expired and blocked the API stop call. The instance was shut down through SSH instead:
sudo shutdown -h now
The SSH session closed and a later SSH check to the last public IP
3.236.132.219 timed out. Refresh AWS SSO and confirm the EC2 state before any
future GPU work:
aws sso login --profile playaround
aws ec2 describe-instances \
--profile playaround \
--region us-east-1 \
--instance-ids i-0fa19dff7b233a353 \
--query 'Reservations[0].Instances[0].State.Name' \
--output text
For future runs, do not stop the instance while a PID is still making progress. Stop it as soon as the run completes, fails, or no longer needs the GPU:
aws ec2 stop-instances \
--profile playaround \
--region us-east-1 \
--instance-ids i-0fa19dff7b233a353