| --- |
| license: apache-2.0 |
| library_name: mlx |
| pipeline_tag: object-detection |
| tags: |
| - document-layout |
| - layout-detection |
| - rt-detr |
| - deformable-detr |
| - paddleocr |
| - pp-doclayout |
| - mlx |
| - apple-silicon |
| --- |
| |
| # PP-DocLayoutV3 — MLX |
|
|
| Apple **MLX** port of **PP-DocLayoutV3**, the RT-DETR-style document-layout |
| detector used by [model-pp-doclayoutv3](https://github.com/agentable/model-pp-doclayoutv3) |
| (image → 25-class layout regions with boxes, instance masks, and reading order). |
| A single `PP-DocLayoutV3.safetensors` (~142 MB fp32), **self-converted from the |
| ONNX graph** ([Bei0001/PP-DocLayoutV3-ONNX](https://huggingface.co/Bei0001/PP-DocLayoutV3-ONNX) |
| — the only export keeping all 4 heads) — no retraining, only a layout conversion |
| gated on numerical parity. The file also carries the AIFI positional embedding |
| and the baked decoder constants (order masks, coordinate grids) the MLX forward |
| needs. |
|
|
| ## Architecture |
|
|
| RT-DETR family: **HGNetV2** CNN backbone → **hybrid encoder** (AIFI transformer |
| layer + CCFM FPN/PAN fusion + mask-feature head) → **6-layer transformer decoder** |
| with **multi-scale deformable attention** (300 queries) → 4 heads: |
|
|
| - `logits` (1,300,25) — class scores |
| - `pred_boxes` (1,300,4) — DETR cxcywh |
| - `out_masks` (1,300,200,200) — instance masks |
| - `order_logits` (1,300,300) — reading order |
|
|
| The MLX backend replays the whole ONNX graph faithfully op-by-op (convs via an |
| NHWC transpose-dance; the deformable GridSample, TopK query selection, GatherND, |
| and bilinear Resize all in MLX), so no part of the intricate MSDeformAttn / box |
| decode is re-derived by hand. |
|
|
| ## Parity |
|
|
| The full forward matches the fp32 ONNX outputs (relative error vs each output's |
| own scale — the order/mask heads carry ±10000 mask-fill magnitudes): |
|
|
| | output | relative | |
| |---|---| |
| | logits | 1.6e-5 | |
| | pred_boxes | 2.2e-5 | |
| | out_masks | 1.3e-5 | |
| | order_logits | 3.2e-6 | |
| |
| End-to-end, the full `Detect` pipeline (preprocess → forward → DB-style postproc) |
| produces **byte-identical layout elements** (class / box / reading order) to the |
| ONNX backend on a real document. |
| |
| ## Usage |
| |
| Backs the MLX backend of `github.com/agentable/model-pp-doclayoutv3`. Build with |
| `-tags mlx` on Apple Silicon and place `PP-DocLayoutV3.safetensors` next to the |
| ONNX graph in the model directory. The default build stays pure-ONNX (no |
| MLX/Metal links). |
| |
| ## License |
| |
| Apache-2.0, following upstream |
| [PaddleOCR / PaddleX](https://github.com/PaddlePaddle/PaddleOCR). |
| |
