prosperious's picture
Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified
|
Raw History Blame Contribute Delete
8.39 kB

PaddleOCR-Compatible Recognizer Training on Apple MPS

Status: verified one-step training evidence on mbp2 and k2 as of 2026-08-09.

This project can train the PP-OCRv5 mobile recognizer's CTC path on Apple GPUs. It does so with a small PyTorch adapter that preserves the PaddleOCR model architecture, not with the stock PaddlePaddle trainer. That distinction matters: the data, dictionary, input profile, and architecture remain PaddleOCR-shaped, but the resulting checkpoint is a PyTorch checkpoint and cannot be loaded into PaddleOCR as a .pdparams file.

Decision

Concern Current path Evidence status
Recognizer model and optimizer PyTorch MPS Forward, backward, update, and checkpoint verified
CTC log probabilities PyTorch MPS Verified
CTC loss CPU with autograd back to MPS Verified; this is an intentional device boundary
Official PaddleOCR training program Stock PaddlePaddle No stock MPS device on the tested runtime
Paddle's custom MPS plugin PaddleCustomDevice prototype Not yet sufficient for PP-OCRv5 training
Checkpoint format PyTorch state_dict Adapter-only until a converter and parity test exist

Use this path for recognizer experiments on the available Macs. Keep the existing Paddle-first page pipeline and its data/evaluation contracts. If an official Paddle checkpoint is a delivery requirement, train on a supported Paddle device or build and validate a deliberate checkpoint converter.

Why Stock Paddle Is Not the Immediate MPS Path

A direct probe on k2 with PaddlePaddle 3.3.1 returned no custom devices and rejected mps, metal, and custom_device:metal device selections. This matches Paddle's macOS CPU build documentation and the PaddleOCR recognition training guide, which does not support macOS GPU training through the stock runtime. The documented paddle.device.set_device forms also do not include a stock MPS target.

Paddle does have an official MPS custom-device backend. Its current build identifies itself as version 0.0.1, requires a signed local build, and its kernel tree covers only a small operator subset such as activation, elementwise, matmul, reshape, softmax, squeeze, and unsqueeze. Its README proves the prototype can expose an mps device and run a simple softmax. It does not yet establish the convolution, normalization, sequence, loss, optimizer, and autograd coverage required by PP-OCRv5. Treat it as the longer-term native-Paddle route, not the current training backend.

The Practical Adapter

The vendored subset comes from the Apache-2.0 frotms/PaddleOCR2Pytorch revision c05307f. It contains the PP-OCRv5 mobile recognition components needed here:

  • PPLCNetV3 at scale 0.95;
  • the SVTR encoder in the MultiHead CTC branch;
  • the CTC head used for this training path.

The NRTR auxiliary branch is not part of the current loss. Its parameters are frozen in the verified smoke run. This is therefore a PP-OCRv5-compatible CTC training path, not a claim of complete reproduction of PaddleOCR's multi-loss recipe.

Three local changes make dynamic-width MPS training work:

  1. The backbone pools height to one while retaining recognition width instead of forcing the original fixed training width. The verified time-axis rule is T = input_width / 4: width 160 produces 40 CTC steps and width 320 produces
  2. A localized contiguity hook before ctc_encoder.encoder.conv3 prevents the MPS convolution from receiving the non-contiguous layout emitted by the preceding SVTR operations.
  3. The model, images, SVTR/CTC logits, and log-softmax remain on MPS. Only CTCLoss runs on CPU; PyTorch autograd carries its gradient back to the MPS graph, after which the optimizer updates MPS parameters.

The CPU loss boundary is a correctness solution, not a throughput claim. It adds device transfers and still needs multi-epoch profiling on real manifests.

Verified Smoke Evidence

The direct checks below exercised the same model shape and completed a finite gradient update. Their elapsed values include cold-start effects and are diagnostics, not benchmarks.

Host Runtime Input -> logits Loss Tracked max parameter delta Cold elapsed
mbp2 (M4 Pro, 20-core GPU, 24 GB) Torch 2.13, MPS [1,3,48,320] -> [1,80,48] about 55.91 0.000501692 about 3.96 s
k2 (M2 Ultra, 60-core GPU, 64 GB) Torch 2.12, MPS [1,3,48,320] -> [1,80,48] 55.94594 0.000501603 9.871 s

An end-to-end adapter smoke on mbp2 used Python 3.11.15 and Torch 2.13.0. Four synthetic samples covered Latin, Devanagari, and Limbu characters. Input [4,3,48,160] produced MPS logits [4,40,7]; CPU CTC loss was 29.883989334106445; all 582 gradient tensors were finite on mps:0; and an optimizer step changed the tracked parameter by 0.000500023365020752. The step took 7.3767 seconds and the sampled MPS current allocation after the step was 226,097,408 bytes. Neither number is a sustained-throughput or peak-memory measurement.

The ephemeral evidence artifacts on mbp2 were:

  • /tmp/paddle2torch-mps-smoke/artifacts/result.json, SHA-256 44427429b01ef308a386bc2cbbab415ee11e78a09262213cee7d0c600ffd1c48;
  • /tmp/paddle2torch-mps-smoke/artifacts/ppocrv5_mobile_ctc_mps_step.pt, SHA-256 0113bb40a024bd88609c529a51b39a88f55f80a8cb4173d110382be162f148ca.

/tmp is not durable storage. A real run must write its config, manifest fingerprint, source revision, environment report, metrics, and checkpoint to a versioned run directory.

Checkpoint Compatibility Boundary

The adapter checkpoint records PyTorch tensor names, optimizer state, and the script-specific CTC head. Do not rename it to .pdparams or pass it to the official PaddleOCR trainer. Likewise, a Paddle checkpoint is not a direct warm start for this path merely because the architecture names match.

Any conversion must explicitly map parameter names and layouts, handle the dictionary-dependent CTC output size, document the local dynamic-pooling change, and prove inference parity on a frozen input set. Until that work exists, compatibility means shared architecture intent, input geometry, dictionary, and datasets—not shared checkpoint files or runtime behavior.

Using mbp2 and k2

The two Macs improve experiment throughput by running independent jobs. They should not be treated as one distributed MPS training job: this project has no validated multi-host MPS DDP path.

  • Use immutable dataset/config fingerprints on both hosts, but give every job a distinct run directory, seed, and experiment ID.
  • Use mbp2 for smoke runs and focused ablations. Its 24 GB unified memory and existing memory pressure make conservative batch sizes important.
  • Use k2 for memory-heavier or longer candidates; its 64 GB unified memory is the more useful capacity advantage.
  • Collect checkpoints and canonical prediction files after each independent run, then compare them with tools/eval_metrics.py. Paddle norm_edit_dis is not CER, and silver labels remain diagnostic rather than claim-grade evidence.

GPU access is useful now and the available MPS machines are sufficient to start architecture and training experiments. The immediate proof still needed is a multi-epoch real-manifest run with checkpoint resume/reload, stable memory, canonical held-out CER, and comparison against the existing Paddle-trained recognizers. An NVIDIA GPU is not required for this adapter path; it remains useful only when native Paddle training or CUDA-specific parity is required.