Image Classification
Py-Feat
facial-expression-analysis
action-units
emotion-recognition
gaze-estimation
face-landmarks
head-pose
blendshapes
multitask
Instructions to use py-feat/face_multitask_v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Py-Feat
How to use py-feat/face_multitask_v2 with Py-Feat:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -21,13 +21,16 @@ A single multi-task convolutional model for facial behavior analysis, used by
|
|
| 21 |
[py-feat](https://github.com/cosanlab/py-feat)'s `Detectorv2`. From one face crop
|
| 22 |
it jointly predicts **action units, categorical emotion, valence/arousal,
|
| 23 |
eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit
|
| 24 |
-
blendshapes**.
|
| 25 |
|
| 26 |
- **Backbone:** ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained)
|
| 27 |
-
- **Heads:** AU graph (AFG/FGG/SC) + unified-feature emotion/V-A
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
## Outputs
|
| 33 |
|
|
@@ -42,21 +45,32 @@ blendshapes**.
|
|
| 42 |
| 68 landmarks | derived | dlib-68 subset sampled from the 478 mesh |
|
| 43 |
| Blendshapes | 52 coefficients [0,1] | MediaPipe/ARKit standard names (browInnerUp, jawOpen, mouthSmileLeft, …) |
|
| 44 |
|
| 45 |
-
## Benchmarks (held-out, file-verified — v2.
|
| 46 |
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
| AU | DISFA+ (12-AU, Cheong protocol) | macro-F1 | **0.693** |
|
| 50 |
-
| AU | DISFA+ (8-AU subset) | macro-F1 | **0.740** |
|
| 51 |
-
| Emotion | RAF-DB official test (7-cls) | acc / macro-F1 | **0.910 / 0.885** |
|
| 52 |
-
| Emotion | AffectNet val (7-cls, drop Contempt) | acc / macro-F1 | **0.616 / 0.612** |
|
| 53 |
-
| Valence/Arousal | Aff-Wild2 official validation | CCC (V / A) | **0.852 / 0.799** |
|
| 54 |
-
| Gaze | MPIIGaze (leave-subject-out) | mean angular err | 7.05° |
|
| 55 |
-
| Gaze | Gaze360 (held-out split) | mean angular err | 12.89° |
|
| 56 |
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
## Usage
|
| 62 |
|
|
@@ -68,7 +82,7 @@ fex = detector.detect("image.jpg") # returns a py-feat Fex
|
|
| 68 |
|
| 69 |
The model expects a face crop produced by RetinaFace + py-feat's
|
| 70 |
`extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2)`,
|
| 71 |
-
then
|
| 72 |
|
| 73 |
## License
|
| 74 |
|
|
|
|
| 21 |
[py-feat](https://github.com/cosanlab/py-feat)'s `Detectorv2`. From one face crop
|
| 22 |
it jointly predicts **action units, categorical emotion, valence/arousal,
|
| 23 |
eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit
|
| 24 |
+
blendshapes** (the v2.6 model; replaces v2.5).
|
| 25 |
|
| 26 |
- **Backbone:** ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained)
|
| 27 |
+
- **Heads:** ME-GraphAU AU graph (AFG/FGG/SC) + unified-feature emotion/V-A heads
|
| 28 |
+
+ landmark, pose, and **blendshape** regression heads + the v2.6 **eye-aware
|
| 29 |
+
gaze head**: RoI-pooled eye features (localized by the predicted mesh),
|
| 30 |
+
conditioned on predicted head pose (6D), with an L2CS-style binned prediction
|
| 31 |
+
over the full ±180° range
|
| 32 |
+
- **Params:** ~41M · **Input:** 224×224 RGB (from a 256×256 face crop)
|
| 33 |
+
- **File:** `face_multitask_v26.safetensors` (safetensors; `ModelV2Config` JSON in the file metadata)
|
| 34 |
|
| 35 |
## Outputs
|
| 36 |
|
|
|
|
| 45 |
| 68 landmarks | derived | dlib-68 subset sampled from the 478 mesh |
|
| 46 |
| Blendshapes | 52 coefficients [0,1] | MediaPipe/ARKit standard names (browInnerUp, jawOpen, mouthSmileLeft, …) |
|
| 47 |
|
| 48 |
+
## Benchmarks (held-out, file-verified — v2.6 deployed checkpoint)
|
| 49 |
|
| 50 |
+
All gaze splits are identity-disjoint from training (held-out subjects), and
|
| 51 |
+
EYEDIAP is never trained on by any py-feat model.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
|
| 53 |
+
| Task | Dataset | Metric | v2.6 | v2.5 |
|
| 54 |
+
|---|---|---|---|---|
|
| 55 |
+
| AU | DISFA+ (12-AU, Cheong protocol) | macro-F1 | **0.696** | 0.693 |
|
| 56 |
+
| AU | DISFA+ (8-AU subset) | macro-F1 | 0.738 | **0.740** |
|
| 57 |
+
| Emotion | AffectNet val (7-cls, drop Contempt) | acc / macro-F1 | 0.615 / 0.610 | **0.616 / 0.612** |
|
| 58 |
+
| Valence/Arousal | AffectNet val | CCC (V / A) | 0.775 / **0.653** | **0.780** / 0.646 |
|
| 59 |
+
| Gaze | ETH-XGaze (held-out subjects) | mean angular err | **5.0°** | 43.2° |
|
| 60 |
+
| Gaze | EYEDIAP (never-train, 15.2K frames) | mean angular err | **13.4°** | 15.3° |
|
| 61 |
+
| Gaze | Gaze360 (held-out split) | mean angular err | 13.0° | **12.9°** |
|
| 62 |
+
| Gaze | MPIIGaze (leave-subject-out) | mean angular err | 7.4° | **7.0°** |
|
| 63 |
+
| Gaze | Columbia (held-out subjects) | mean angular err | **5.4°** | — (trained) |
|
| 64 |
+
|
| 65 |
+
Notes: **v2.6 = v2.5 + an eye-aware gaze head** (eye RoI features, head-pose
|
| 66 |
+
conditioning, binned ±180° prediction) trained with ETH-XGaze added to the gaze
|
| 67 |
+
pool. It transforms extreme-head-pose gaze (ETH-XGaze 43°→5°) and
|
| 68 |
+
out-of-distribution gaze (EYEDIAP −2°), decouples eye gaze from head pose
|
| 69 |
+
(v2.5's gaze tracked the head; v2.6 tracks the eyes), and holds AU / emotion /
|
| 70 |
+
valence-arousal at v2.5 levels within noise. The small MPII / Gaze360 deltas
|
| 71 |
+
(+0.1–0.4°) are the cost of the pose-robust training mix. Numbers are from the
|
| 72 |
+
deployed checkpoint (v2.6 stage-3 `v24_best`, ep07), weight-verified against
|
| 73 |
+
the published `.safetensors`.
|
| 74 |
|
| 75 |
## Usage
|
| 76 |
|
|
|
|
| 82 |
|
| 83 |
The model expects a face crop produced by RetinaFace + py-feat's
|
| 84 |
`extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2)`,
|
| 85 |
+
then resized to 224 and ImageNet-normalized. `Detectorv2` handles this.
|
| 86 |
|
| 87 |
## License
|
| 88 |
|