README.md

---
library_name: py-feat
tags:
- face
- mediapipe
- pls
---

# blendshape + pose -> MP 478 mesh PLS (v5)

![Each blendshape activated, tessellated MediaPipe mesh colored by per-vertex displacement from neutral.](bs_to_mesh_montage.png)

Full-rank linear regression mapping 52 blendshape features + 3 pose covariates
(Pitch, Yaw, Roll) to 478×3 = 1434 MediaPipe FaceMesh vertex coordinates in a
pose-canonical (Procrustes-aligned) frame. Companion to `au_to_mesh_pls` — same
recipe and SHARED mesh frame, so AU / emotion / blendshape mesh predictions are
mutually consistent for visualization.

## Training data
- 633,207 frames from 34,854 CelebV-HQ celebrity videos
- All predictions from the SAME forward on the CelebV-HQ chips:
  mesh478 + 52 blendshapes (MPDetector), AU + emotion + pose (Detectorv2 v2.x)
- Pose-filtered to |yaw| <= 40°, |pitch| <= 30°
- Per-frame Umeyama similarity Procrustes to a GPA reference of 12 stable
  upper-face anchors (forehead 10/9/8/151, nose bridge 6/168/197/195,
  outer canthi 33/263, inner canthi 133/362). Removes (R, s, t).
- Top 1% of frames by max anchor residual dropped.
- Absolute aligned coords — NO per-subject neutral subtraction (Cheong /
  py-feat tutorial-06 recipe).

## Method
- Full-rank linear (== full-rank PLSRegression(scale=True); verified equal in
  prediction on a held-out subsample). Per user request, no low-rank truncation.
- Training inputs: [52 blendshape | 3 pose | 156 blendshape×pose] features.
- Deployed inputs: [52 blendshape | 3 pose] (interactions vanish at pose=0).
- Outputs: 1434-d absolute pose-canonical mesh coords, axis-major [x|y|z].

## Out-of-sample performance (3-fold GroupKFold by video)
- Variance-weighted R² = 0.6389 ± 0.0028
- Per-fold R² = [0.6402, 0.6350, 0.6415]
- MAE = 1.2945 (au_to_mesh v5 canonical-frame units)

## Inference
```python
import numpy as np
m = np.load("bs_to_mesh_pls_v5.npz")
f = np.zeros(len(m["feature_columns"]))
# e.g. f[m["feature_columns"].tolist().index("happiness")] = 1.0
pose = np.zeros(3)                                  # [Pitch, Yaw, Roll]
x = np.concatenate([f, pose])                       # (55,)
flat = x @ m["coef"] + m["intercept"]               # (1434,)
mesh = np.stack([flat[:478], flat[478:956], flat[956:]], axis=1)  # (478,3) axis-major
```

## File format
NPZ: coef (55, 1434) f32; intercept (1434,) f32; input_columns; feature_columns;
feature_name; pose_columns; mean_aligned_mesh (478,3); mean_neutral_mesh (478,3);
reference_anchors (12,3); anchor_indices (12,); n_components (); model_card; training_metadata (JSON).