| --- |
| license: apache-2.0 |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - keypoint-detection |
| base_model: |
| - usyd-community/vitpose-plus-base |
| --- |
| # ViTPose+ base — ExecuTorch (heatmap pose) |
|
|
| A person crop in, seventeen COCO keypoints out, from a plain ViT. This is the |
| heatmap contract — the one most pose code on the internet expects — where the |
| shelf's other pose models (RTMPose, RTMW) emit SimCC distributions instead. |
|
|
| ``` |
| pixel_values (1,3,256,192) -> heatmaps (1,17,64,48) |
| ``` |
|
|
| | build | file | MB | corr vs fp32 eager | Mac ms* | |
| |---|---|---|---|---| |
| | fp32 | `vitpose_plus_base_xnnpack_fp32.pte` | 501.9 | 1.000000 | 60.4 | |
| | Core ML (fp16, iOS) | `vitpose_plus_base_coreml_all.pte` | 251.6 | 0.999998 | 7.1 | |
|
|
| \*Mac arm64, single process, median of 10 — a reference point for relative cost, |
| not a device number. Torch eager fp32 on the same machine: 48.8 ms. XNNPACK |
| delegate coverage 65.1% (630/967 ops); the Core ML build is 100% delegated in one |
| subgraph. |
| |
| ## Running it |
| |
| **1. The crop.** One person, RGB, divided by 255, ImageNet normalised (mean |
| .485/.456/.406, std .229/.224/.225), resized to **256 high by 192 wide**. Bring |
| your own detector — this model does not find people, it reads the pose of the one |
| you hand it. |
| |
| **2. The decode.** Joint `k` is the argmax of heatmap channel `k`. With a 64×48 |
| heatmap over a 256×192 crop, the scale factor is 4: |
| |
| ``` |
| i = argmax(heatmaps[0, k]) # over the flattened 64x48 grid |
| x = (i % 48) * 4 # crop pixels |
| y = (i // 48) * 4 |
| confidence = heatmaps[0, k].max() |
| ``` |
| |
| Keypoints come in COCO order: nose, eyes, ears, shoulders, elbows, wrists, hips, |
| knees, ankles. Map back to the original image with the box you cropped from. |
| |
| A refinement worth knowing about: the reference implementation takes a weighted |
| average around the peak instead of the raw argmax, which buys sub-pixel accuracy. |
| The recipe above is the plain one, and it is what the checks below measure. |
| |
| ## The expert index |
| |
| The "+" checkpoint carries a mixture of experts, chosen by a `dataset_index` |
| argument — the model can speak COCO, AI Challenger, MPII, AP-10K, APT-36K or |
| whole-body. **This graph fixes it at 0, the COCO expert, and does not take the |
| index.** It selects a keypoint convention rather than a per-call option, and an |
| app that passed 5 would get a whole-body skeleton decoded as if it were COCO. |
| |
| Measured on six person crops, asking each expert in turn: indices 0–3 all produce |
| anatomically ordered skeletons, index 4 five of six, and index 5 none — which is |
| what the documented order predicts, whole-body being a different convention. |
| |
| ## What is measured |
| |
| The card's own decode, run on six person crops: **6/6 skeletons in anatomical |
| order** — nose above shoulders above hips — with every keypoint inside the crop. |
| |
| ```bash |
| python convert/verify_cards.py vitpose_plus_base |
| python convert/audit_int8.py vitpose_plus_base |
| ``` |
| |
| ## Not shipped, and why |
| |
| **int8 converts and is not published.** It is 141.3 MB against fp32's 501.9 MB at |
| correlation 0.999904, and it passes the anatomical check 6/6. The number that |
| decides is where the keypoints land: on ten real images, the fraction of confident |
| joints within 4 px of the fp32 build's is **1.0000 median and 0.8235 at worst** — |
| three joints out of seventeen moving further than 4 px on one image. This shelf |
| gates pose on the worst image, not the median, because a build that is perfect on |
| nine photographs and visibly wrong on the tenth is what a user finds. The bar is |
| 0.90. |
| |
| Correlation would have cleared it comfortably, which is the reason the bar is not |
| correlation. |
| |
| - **Source**: [usyd-community/vitpose-plus-base](https://huggingface.co/usyd-community/vitpose-plus-base) |
| - **License**: Apache-2.0 |
| |
| torch.export -> to_edge_transform_and_lower(partitioner) -> .pte |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models)) |
| |
