README.md

---
license: other
license_name: research-only
license_link: LICENSE
library_name: py-feat
tags:
  - facial-expression-analysis
  - action-units
  - emotion-recognition
  - gaze-estimation
  - face-landmarks
  - head-pose
  - blendshapes
  - multitask
pipeline_tag: image-classification
---

# face_multitask_v2

A single multi-task convolutional model for facial behavior analysis, used by
[py-feat](https://github.com/cosanlab/py-feat)'s `Detectorv2`. From one face crop
it jointly predicts **action units, categorical emotion, valence/arousal,
eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit
blendshapes**.

- **Backbone:** ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained)
- **Heads:** AU graph (AFG/FGG/SC) + unified-feature emotion/V-A and
  gaze heads + landmark, pose, and **blendshape** regression heads
- **Params:** ~30M · **Input:** 224×224 RGB (from a 256×256 face crop)
- **File:** `face_multitask_v2.safetensors` (safetensors; `ModelV2Config` JSON in the file metadata)

## Outputs

| Task | Output | Notes |
|---|---|---|
| Action Units | 20 probabilities [0,1] | AU01,02,04,05,06,07,09,10,11,12,14,15,17,20,23,24,25,26,28,43 |
| Emotion | 7-class softmax | Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger |
| Valence / Arousal | 2 × [−1,1] | tanh |
| Gaze | (yaw, pitch) radians | head-centric; yaw+ = right, pitch+ = up |
| Face mesh | 478 × (x,y,z) | MediaPipe topology, chip-pixel coords (z = relative depth) |
| Head pose | (yaw, pitch, roll, tx, ty, tz) | radians / pixels |
| 68 landmarks | derived | dlib-68 subset sampled from the 478 mesh |
| Blendshapes | 52 coefficients [0,1] | MediaPipe/ARKit standard names (browInnerUp, jawOpen, mouthSmileLeft, …) |

## Benchmarks (held-out, file-verified — v2.5 deployed checkpoint)

| Task | Dataset | Metric | Score |
|---|---|---|---|
| AU | DISFA+ (12-AU, Cheong protocol) | macro-F1 | **0.693** |
| AU | DISFA+ (8-AU subset) | macro-F1 | **0.740** |
| Emotion | RAF-DB official test (7-cls) | acc / macro-F1 | **0.910 / 0.885** |
| Emotion | AffectNet val (7-cls, drop Contempt) | acc / macro-F1 | **0.616 / 0.612** |
| Valence/Arousal | Aff-Wild2 official validation | CCC (V / A) | **0.852 / 0.799** |
| Gaze | MPIIGaze (leave-subject-out) | mean angular err | 7.05° |
| Gaze | Gaze360 (held-out split) | mean angular err | 12.89° |

Notes: **Gaze numbers are now leave-subject-out
held-out** (honest generalization); Numbers are from the deployed checkpoint
(`v25c_release_ep14`), weight-verified against the published `.safetensors`.

## Usage

```python
from feat import Detectorv2
detector = Detectorv2(device="cuda")
fex = detector.detect("image.jpg")   # returns a py-feat Fex
```

The model expects a face crop produced by RetinaFace + py-feat's
`extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2)`,
then center-cropped to 224 and ImageNet-normalized. `Detectorv2` handles this.

## License

**Research / non-commercial use only.** Trained on datasets (AffectNet, DISFA+,
RAF-DB, Aff-Wild2, BP4D, etc.) whose licenses restrict use to academic research.
The ConvNeXt-V2 backbone is MIT-licensed. Confirm each constituent dataset's
terms before any non-research use.