README.md

---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- keypoint-detection
base_model:
- usyd-community/vitpose-plus-base
---
# ViTPose+ base — ExecuTorch (heatmap pose)

A person crop in, seventeen COCO keypoints out, from a plain ViT. This is the
heatmap contract — the one most pose code on the internet expects — where the
shelf's other pose models (RTMPose, RTMW) emit SimCC distributions instead.

```
pixel_values (1,3,256,192) -> heatmaps (1,17,64,48)
```

| build | file | MB | corr vs fp32 eager | Mac ms* |
|---|---|---|---|---|
| fp32 | `vitpose_plus_base_xnnpack_fp32.pte` | 501.9 | 1.000000 | 60.4 |
| Core ML (fp16, iOS) | `vitpose_plus_base_coreml_all.pte` | 251.6 | 0.999998 | 7.1 |

\*Mac arm64, single process, median of 10 — a reference point for relative cost,
not a device number. Torch eager fp32 on the same machine: 48.8 ms. XNNPACK
delegate coverage 65.1% (630/967 ops); the Core ML build is 100% delegated in one
subgraph.

## Running it

**1. The crop.** One person, RGB, divided by 255, ImageNet normalised (mean
.485/.456/.406, std .229/.224/.225), resized to **256 high by 192 wide**. Bring
your own detector — this model does not find people, it reads the pose of the one
you hand it.

**2. The decode.** Joint `k` is the argmax of heatmap channel `k`. With a 64×48
heatmap over a 256×192 crop, the scale factor is 4:

```
i = argmax(heatmaps[0, k])        # over the flattened 64x48 grid
x = (i % 48) * 4                  # crop pixels
y = (i // 48) * 4
confidence = heatmaps[0, k].max()
```

Keypoints come in COCO order: nose, eyes, ears, shoulders, elbows, wrists, hips,
knees, ankles. Map back to the original image with the box you cropped from.

A refinement worth knowing about: the reference implementation takes a weighted
average around the peak instead of the raw argmax, which buys sub-pixel accuracy.
The recipe above is the plain one, and it is what the checks below measure.

## The expert index

The "+" checkpoint carries a mixture of experts, chosen by a `dataset_index`
argument — the model can speak COCO, AI Challenger, MPII, AP-10K, APT-36K or
whole-body. **This graph fixes it at 0, the COCO expert, and does not take the
index.** It selects a keypoint convention rather than a per-call option, and an
app that passed 5 would get a whole-body skeleton decoded as if it were COCO.

Measured on six person crops, asking each expert in turn: indices 0–3 all produce
anatomically ordered skeletons, index 4 five of six, and index 5 none — which is
what the documented order predicts, whole-body being a different convention.

## What is measured

The card's own decode, run on six person crops: **6/6 skeletons in anatomical
order** — nose above shoulders above hips — with every keypoint inside the crop.

```bash
python convert/verify_cards.py vitpose_plus_base
python convert/audit_int8.py vitpose_plus_base
```

## Not shipped, and why

**int8 converts and is not published.** It is 141.3 MB against fp32's 501.9 MB at
correlation 0.999904, and it passes the anatomical check 6/6. The number that
decides is where the keypoints land: on ten real images, the fraction of confident
joints within 4 px of the fp32 build's is **1.0000 median and 0.8235 at worst** —
three joints out of seventeen moving further than 4 px on one image. This shelf
gates pose on the worst image, not the median, because a build that is perfect on
nine photographs and visibly wrong on the tenth is what a user finds. The bar is
0.90.

Correlation would have cleared it comfortably, which is the reason the bar is not
correlation.

- **Source**: [usyd-community/vitpose-plus-base](https://huggingface.co/usyd-community/vitpose-plus-base)
- **License**: Apache-2.0

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))