ljchang commited on
Commit
e0a6fba
·
verified ·
1 Parent(s): 1c55f90

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +33 -19
README.md CHANGED
@@ -21,13 +21,16 @@ A single multi-task convolutional model for facial behavior analysis, used by
21
  [py-feat](https://github.com/cosanlab/py-feat)'s `Detectorv2`. From one face crop
22
  it jointly predicts **action units, categorical emotion, valence/arousal,
23
  eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit
24
- blendshapes**.
25
 
26
  - **Backbone:** ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained)
27
- - **Heads:** AU graph (AFG/FGG/SC) + unified-feature emotion/V-A and
28
- gaze heads + landmark, pose, and **blendshape** regression heads
29
- - **Params:** ~30M · **Input:** 224×224 RGB (from a 256×256 face crop)
30
- - **File:** `face_multitask_v2.safetensors` (safetensors; `ModelV2Config` JSON in the file metadata)
 
 
 
31
 
32
  ## Outputs
33
 
@@ -42,21 +45,32 @@ blendshapes**.
42
  | 68 landmarks | derived | dlib-68 subset sampled from the 478 mesh |
43
  | Blendshapes | 52 coefficients [0,1] | MediaPipe/ARKit standard names (browInnerUp, jawOpen, mouthSmileLeft, …) |
44
 
45
- ## Benchmarks (held-out, file-verified — v2.5 deployed checkpoint)
46
 
47
- | Task | Dataset | Metric | Score |
48
- |---|---|---|---|
49
- | AU | DISFA+ (12-AU, Cheong protocol) | macro-F1 | **0.693** |
50
- | AU | DISFA+ (8-AU subset) | macro-F1 | **0.740** |
51
- | Emotion | RAF-DB official test (7-cls) | acc / macro-F1 | **0.910 / 0.885** |
52
- | Emotion | AffectNet val (7-cls, drop Contempt) | acc / macro-F1 | **0.616 / 0.612** |
53
- | Valence/Arousal | Aff-Wild2 official validation | CCC (V / A) | **0.852 / 0.799** |
54
- | Gaze | MPIIGaze (leave-subject-out) | mean angular err | 7.05° |
55
- | Gaze | Gaze360 (held-out split) | mean angular err | 12.89° |
56
 
57
- Notes: **Gaze numbers are now leave-subject-out
58
- held-out** (honest generalization); Numbers are from the deployed checkpoint
59
- (`v25c_release_ep14`), weight-verified against the published `.safetensors`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
60
 
61
  ## Usage
62
 
@@ -68,7 +82,7 @@ fex = detector.detect("image.jpg") # returns a py-feat Fex
68
 
69
  The model expects a face crop produced by RetinaFace + py-feat's
70
  `extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2)`,
71
- then center-cropped to 224 and ImageNet-normalized. `Detectorv2` handles this.
72
 
73
  ## License
74
 
 
21
  [py-feat](https://github.com/cosanlab/py-feat)'s `Detectorv2`. From one face crop
22
  it jointly predicts **action units, categorical emotion, valence/arousal,
23
  eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit
24
+ blendshapes** (the v2.6 model; replaces v2.5).
25
 
26
  - **Backbone:** ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained)
27
+ - **Heads:** ME-GraphAU AU graph (AFG/FGG/SC) + unified-feature emotion/V-A heads
28
+ + landmark, pose, and **blendshape** regression heads + the v2.6 **eye-aware
29
+ gaze head**: RoI-pooled eye features (localized by the predicted mesh),
30
+ conditioned on predicted head pose (6D), with an L2CS-style binned prediction
31
+ over the full ±180° range
32
+ - **Params:** ~41M · **Input:** 224×224 RGB (from a 256×256 face crop)
33
+ - **File:** `face_multitask_v26.safetensors` (safetensors; `ModelV2Config` JSON in the file metadata)
34
 
35
  ## Outputs
36
 
 
45
  | 68 landmarks | derived | dlib-68 subset sampled from the 478 mesh |
46
  | Blendshapes | 52 coefficients [0,1] | MediaPipe/ARKit standard names (browInnerUp, jawOpen, mouthSmileLeft, …) |
47
 
48
+ ## Benchmarks (held-out, file-verified — v2.6 deployed checkpoint)
49
 
50
+ All gaze splits are identity-disjoint from training (held-out subjects), and
51
+ EYEDIAP is never trained on by any py-feat model.
 
 
 
 
 
 
 
52
 
53
+ | Task | Dataset | Metric | v2.6 | v2.5 |
54
+ |---|---|---|---|---|
55
+ | AU | DISFA+ (12-AU, Cheong protocol) | macro-F1 | **0.696** | 0.693 |
56
+ | AU | DISFA+ (8-AU subset) | macro-F1 | 0.738 | **0.740** |
57
+ | Emotion | AffectNet val (7-cls, drop Contempt) | acc / macro-F1 | 0.615 / 0.610 | **0.616 / 0.612** |
58
+ | Valence/Arousal | AffectNet val | CCC (V / A) | 0.775 / **0.653** | **0.780** / 0.646 |
59
+ | Gaze | ETH-XGaze (held-out subjects) | mean angular err | **5.0°** | 43.2° |
60
+ | Gaze | EYEDIAP (never-train, 15.2K frames) | mean angular err | **13.4°** | 15.3° |
61
+ | Gaze | Gaze360 (held-out split) | mean angular err | 13.0° | **12.9°** |
62
+ | Gaze | MPIIGaze (leave-subject-out) | mean angular err | 7.4° | **7.0°** |
63
+ | Gaze | Columbia (held-out subjects) | mean angular err | **5.4°** | — (trained) |
64
+
65
+ Notes: **v2.6 = v2.5 + an eye-aware gaze head** (eye RoI features, head-pose
66
+ conditioning, binned ±180° prediction) trained with ETH-XGaze added to the gaze
67
+ pool. It transforms extreme-head-pose gaze (ETH-XGaze 43°→5°) and
68
+ out-of-distribution gaze (EYEDIAP −2°), decouples eye gaze from head pose
69
+ (v2.5's gaze tracked the head; v2.6 tracks the eyes), and holds AU / emotion /
70
+ valence-arousal at v2.5 levels within noise. The small MPII / Gaze360 deltas
71
+ (+0.1–0.4°) are the cost of the pose-robust training mix. Numbers are from the
72
+ deployed checkpoint (v2.6 stage-3 `v24_best`, ep07), weight-verified against
73
+ the published `.safetensors`.
74
 
75
  ## Usage
76
 
 
82
 
83
  The model expects a face crop produced by RetinaFace + py-feat's
84
  `extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2)`,
85
+ then resized to 224 and ImageNet-normalized. `Detectorv2` handles this.
86
 
87
  ## License
88