license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- keypoint-detection
base_model:
- usyd-community/vitpose-plus-base
ViTPose+ base — ExecuTorch (heatmap pose)
A person crop in, seventeen COCO keypoints out, from a plain ViT. This is the heatmap contract — the one most pose code on the internet expects — where the shelf's other pose models (RTMPose, RTMW) emit SimCC distributions instead.
pixel_values (1,3,256,192) -> heatmaps (1,17,64,48)
| build | file | MB | corr vs fp32 eager | Mac ms* |
|---|---|---|---|---|
| fp32 | vitpose_plus_base_xnnpack_fp32.pte |
501.9 | 1.000000 | 60.4 |
| Core ML (fp16, iOS) | vitpose_plus_base_coreml_all.pte |
251.6 | 0.999998 | 7.1 |
*Mac arm64, single process, median of 10 — a reference point for relative cost, not a device number. Torch eager fp32 on the same machine: 48.8 ms. XNNPACK delegate coverage 65.1% (630/967 ops); the Core ML build is 100% delegated in one subgraph.
Running it
1. The crop. One person, RGB, divided by 255, ImageNet normalised (mean .485/.456/.406, std .229/.224/.225), resized to 256 high by 192 wide. Bring your own detector — this model does not find people, it reads the pose of the one you hand it.
2. The decode. Joint k is the argmax of heatmap channel k. With a 64×48
heatmap over a 256×192 crop, the scale factor is 4:
i = argmax(heatmaps[0, k]) # over the flattened 64x48 grid
x = (i % 48) * 4 # crop pixels
y = (i // 48) * 4
confidence = heatmaps[0, k].max()
Keypoints come in COCO order: nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles. Map back to the original image with the box you cropped from.
A refinement worth knowing about: the reference implementation takes a weighted average around the peak instead of the raw argmax, which buys sub-pixel accuracy. The recipe above is the plain one, and it is what the checks below measure.
The expert index
The "+" checkpoint carries a mixture of experts, chosen by a dataset_index
argument — the model can speak COCO, AI Challenger, MPII, AP-10K, APT-36K or
whole-body. This graph fixes it at 0, the COCO expert, and does not take the
index. It selects a keypoint convention rather than a per-call option, and an
app that passed 5 would get a whole-body skeleton decoded as if it were COCO.
Measured on six person crops, asking each expert in turn: indices 0–3 all produce anatomically ordered skeletons, index 4 five of six, and index 5 none — which is what the documented order predicts, whole-body being a different convention.
What is measured
The card's own decode, run on six person crops: 6/6 skeletons in anatomical order — nose above shoulders above hips — with every keypoint inside the crop.
python convert/verify_cards.py vitpose_plus_base
python convert/audit_int8.py vitpose_plus_base
Not shipped, and why
int8 converts and is not published. It is 141.3 MB against fp32's 501.9 MB at correlation 0.999904, and it passes the anatomical check 6/6. The number that decides is where the keypoints land: on ten real images, the fraction of confident joints within 4 px of the fp32 build's is 1.0000 median and 0.8235 at worst — three joints out of seventeen moving further than 4 px on one image. This shelf gates pose on the worst image, not the median, because a build that is perfect on nine photographs and visibly wrong on the tenth is what a user finds. The bar is 0.90.
Correlation would have cleared it comfortably, which is the reason the bar is not correlation.
- Source: usyd-community/vitpose-plus-base
- License: Apache-2.0
torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: executorch-models)