ljchang commited on
Commit
76398a3
·
verified ·
1 Parent(s): 5e16dbb

Model card: document v2.8 as the current default

Browse files
Files changed (1) hide show
  1. README.md +104 -52
README.md CHANGED
@@ -21,17 +21,20 @@ A single multi-task convolutional model for facial behavior analysis, used by
21
  [py-feat](https://github.com/cosanlab/py-feat)'s `Detectorv2`. From one face crop
22
  it jointly predicts **action units, categorical emotion, valence/arousal,
23
  eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit
24
- blendshapes** (the v2.7 model; replaces v2.6).
 
 
25
 
26
  - **Backbone:** ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained)
27
- - **Heads:** ME-GraphAU AU graph (AFG/FGG/SC) + unified-feature emotion/V-A heads
28
- + landmark, pose, and **blendshape** regression heads + the v2.6 **eye-aware
29
- gaze head**: RoI-pooled eye features (localized by the predicted mesh),
30
- conditioned on predicted head pose (6D), with an L2CS-style binned prediction
31
- over the full ±180° range (v2.7: 2° bins)
32
- - **Params:** ~42M · **Input:** 224×224 RGB (from a 256×256 face crop)
33
- - **File:** `face_multitask_v27.safetensors` (safetensors; `ModelV2Config` JSON in the file metadata)
34
- - **Weights:** soup (equal average) of five consecutive fine-tuning checkpoints
 
35
 
36
  ## Outputs
37
 
@@ -40,47 +43,89 @@ blendshapes** (the v2.7 model; replaces v2.6).
40
  | Action Units | 20 probabilities [0,1] | AU01,02,04,05,06,07,09,10,11,12,14,15,17,20,23,24,25,26,28,43 |
41
  | Emotion | 7-class softmax | Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger |
42
  | Valence / Arousal | 2 × [−1,1] | tanh |
43
- | Gaze | (yaw, pitch) radians | RAW convention is y-down: yaw+ = subject's right (image-left), pitch+ = looking DOWN. `Detectorv2` negates pitch so Fex columns are canonical +up (since py-feat 2.1.1) |
44
  | Face mesh | 478 × (x,y,z) | MediaPipe topology, chip-pixel coords (z = relative depth) |
45
- | Head pose | (pitch, yaw, roll, tx, ty, tz) | radians / pixels; RAW pitch+ = down (img2pose teacher frame); `Detectorv2` outputs canonical +up (since py-feat 2.1.1) |
46
  | 68 landmarks | derived | dlib-68 subset sampled from the 478 mesh |
47
- | Blendshapes | 52 coefficients [0,1] | MediaPipe/ARKit standard names (browInnerUp, jawOpen, mouthSmileLeft, …) |
48
-
49
- ## Benchmarks (held-out, file-verified — v2.7 deployed checkpoint)
50
-
51
- All gaze splits are identity-disjoint from training (held-out subjects), and
52
- EYEDIAP is never trained on by any py-feat model.
53
-
54
- | Task | Dataset | Metric | v2.7 | v2.6 | v2.5 |
55
- |---|---|---|---|---|---|
56
- | AU | DISFA+ (12-AU, Cheong protocol) | macro-F1 | 0.686 | **0.696** | 0.693 |
57
- | AU | DISFA+ (8-AU subset) | macro-F1 | **0.740** | 0.738 | **0.740** |
58
- | Emotion | AffectNet val (7-cls, drop Contempt) | acc / macro-F1 | 0.612 / 0.607 | 0.615 / 0.610 | **0.616 / 0.612** |
59
- | Emotion | RAF-DB test | acc / macro-F1 | 0.876 / 0.817 | 0.873 / 0.818 | **0.910 / 0.885** |
60
- | Valence/Arousal | AffectNet val | CCC (V / A) | 0.773 / 0.647 | 0.775 / **0.653** | **0.780** / 0.646 |
61
- | Valence/Arousal | AFEW-VA | CCC (V / A) | 0.711 / 0.480 | 0.718 / 0.411 | **0.833 / 0.863** |
62
- | Valence/Arousal | Aff-Wild2 val | CCC (V / A) | 0.331 / 0.418 | 0.397 / 0.458 | **0.852 / 0.799** |
63
- | Gaze | ETH-XGaze (held-out subjects) | mean angular err | 5.1° | **5.0°** | 43.2° |
64
- | Gaze | EYEDIAP (never-train, 15.2K frames) | mean angular err | **12.6°** | 13.4° | 15.3° |
65
- | Gaze | Gaze360 (held-out split) | mean angular err | 13.0° | 13.0° | **12.9°** |
66
- | Gaze | MPIIGaze (leave-subject-out) | mean angular err | 8.0° | 7.4° | **7.0°** |
67
- | Gaze | Columbia (held-out subjects) | mean angular err | **4.1°** | 5.4° | — (trained) |
68
- | Blendshapes | FacePlace (teacher agreement) | mean active-ch. r | **0.761** | 0.748 | 0.756 |
69
-
70
- Notes: **v2.7 = the v2.6 architecture retrained** with a rescaled gaze loss,
71
- a rebalanced within-gaze data mix, per-source augmentation, 2° gaze bins, and
72
- a head-pose label fix. Vs v2.6 it improves out-of-distribution gaze (EYEDIAP
73
- −0.7°, Columbia −1.3°), RAF-DB macro-F1 end-to-end (+2.7), AFEW-VA arousal,
74
- blendshape fidelity (best of any release), and occlusion robustness, at the
75
- cost of ~0.6° on frontal MPIIGaze and 0.01 on the 12-AU set.
76
-
77
- **Known limitation (v2.6 and v2.7):** continuous valence/arousal on
78
- *video-frame* corpora (Aff-Wild2, AFEW-VA) is substantially below v2.5
79
- (e.g. Aff-Wild2 CCC-V 0.85 → 0.33). AffectNet (still-image) V/A is unaffected.
80
- If frame-wise continuous V/A on video is your primary measure, prefer the v2.5
81
- weights (`face_multitask_v2.safetensors`, still published in this repo). A fix
82
- is under investigation. Numbers are from the deployed checkpoint (v2.7
83
- stage-3 soup ep05-09), weight-verified against the published `.safetensors`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
84
 
85
  ## Usage
86
 
@@ -90,13 +135,20 @@ detector = Detectorv2(device="cuda")
90
  fex = detector.detect("image.jpg") # returns a py-feat Fex
91
  ```
92
 
 
 
 
 
 
 
 
93
  The model expects a face crop produced by RetinaFace + py-feat's
94
  `extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2)`,
95
- then resized to 224 and ImageNet-normalized. `Detectorv2` handles this.
96
 
97
  ## License
98
 
99
  **Research / non-commercial use only.** Trained on datasets (AffectNet, DISFA+,
100
- RAF-DB, Aff-Wild2, BP4D, etc.) whose licenses restrict use to academic research.
101
- The ConvNeXt-V2 backbone is MIT-licensed. Confirm each constituent dataset's
102
- terms before any non-research use.
 
21
  [py-feat](https://github.com/cosanlab/py-feat)'s `Detectorv2`. From one face crop
22
  it jointly predicts **action units, categorical emotion, valence/arousal,
23
  eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit
24
+ blendshapes**.
25
+
26
+ Current default: **`face_multitask_v28.safetensors`**.
27
 
28
  - **Backbone:** ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained)
29
+ - **Heads:** ME-GraphAU AU graph (AFG/FGG/SC) + unified-feature emotion/V-A and
30
+ gaze heads + landmark, pose, and blendshape regression heads
31
+ - **Params:** ~30M · **Input:** 224×224 RGB (from a 256×256 face crop)
32
+ - **Format:** safetensors, with the `ModelV2Config` JSON in the file metadata
33
+ - **Weights:** uniform weight average ("model soup") of three training epochs
34
+
35
+ Older files in this repo (`face_multitask_v2`, `_v26`, `_v27`) are retained so
36
+ existing installs keep working. Each py-feat release pins the filename it was
37
+ built against — older code cannot construct newer architectures.
38
 
39
  ## Outputs
40
 
 
43
  | Action Units | 20 probabilities [0,1] | AU01,02,04,05,06,07,09,10,11,12,14,15,17,20,23,24,25,26,28,43 |
44
  | Emotion | 7-class softmax | Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger |
45
  | Valence / Arousal | 2 × [−1,1] | tanh |
46
+ | Gaze | (yaw, pitch) radians | head-centric; yaw+ = right, pitch+ = up |
47
  | Face mesh | 478 × (x,y,z) | MediaPipe topology, chip-pixel coords (z = relative depth) |
48
+ | Head pose | (yaw, pitch, roll, tx, ty, tz) | radians / pixels |
49
  | 68 landmarks | derived | dlib-68 subset sampled from the 478 mesh |
50
+ | Blendshapes | 52 coefficients [0,1] | MediaPipe/ARKit standard names |
51
+
52
+ ## Evaluation protocol
53
+
54
+ Every benchmark below is **held out from training at the split level**, and
55
+ DISFA+ additionally at the **identity** level. The training run reserves 16
56
+ splits: `affectnet:val`, `raf_db:test`, `ferplus:{test,val}`,
57
+ `meld:{bench,val}`, `afew_va:val`, `aff_wild2:bench`, `aff_wild2_va:val`,
58
+ `gaze360:{bench,val}`, `mpii_gaze:test`, `mpii_facegaze:test`,
59
+ `columbia_gaze:test`, `ethxgaze:test`, `eyediap:test`. The `disfaplus` and
60
+ `mpii_facegaze` sources are excluded outright, and DISFA+ identities appearing
61
+ in other corpora are excluded as well.
62
+
63
+ ## Benchmarks
64
+
65
+ Chip-protocol inference on held-out splits.
66
+
67
+ | Task | Dataset | Metric | Score |
68
+ |---|---|---|---|
69
+ | AU | DISFA+ (12-AU, Cheong protocol) | macro-F1 | **0.682** |
70
+ | AU | DISFA+ (common-8 subset) | macro-F1 | **0.772** |
71
+ | Emotion | AffectNet val (7-cls, drop Contempt) | acc / macro-F1 | **0.628 / 0.625** |
72
+ | Emotion | RAF-DB official test (7-cls) | acc / macro-F1 | **0.869 / 0.806** |
73
+ | Valence/Arousal | AffectNet val | CCC (V / A) | **0.773 / 0.650** |
74
+ | Valence/Arousal | Aff-Wild2 official validation | CCC (V / A) | **0.376 / 0.477** |
75
+ | Valence/Arousal | AFEW-VA validation | CCC (V / A) | **0.687 / 0.548** |
76
+ | Gaze | Gaze360 (held-out split) | mean angular err | **13.04°** |
77
+ | Gaze | MPIIGaze (leave-subject-out) | mean angular err | **8.26°** |
78
+ | Gaze | ETH-XGaze (test) | mean angular err | **4.76°** |
79
+ | Gaze | Columbia (test) | mean angular err | **3.76°** |
80
+ | Gaze | EYEDIAP (test, never trained on) | mean angular err | **10.61°** |
81
+
82
+ ### Cross-tool, end-to-end
83
+
84
+ Full shipped pipeline (detect → align → predict) on raw frames, scored on the
85
+ images all tools processed. AU presence = DISFA+ intensity ≥ 2, prediction ≥ 0.5.
86
+
87
+ | Benchmark | This model | OpenFace 3.0 | LibreFace |
88
+ |---|---:|---:|---:|
89
+ | DISFA+ AU, common-8 macro-F1 | **0.774** | 0.732 | 0.492 |
90
+ | DISFA+ AU, 12-AU macro-F1 | **0.671** | — (8 AUs only) | 0.397 |
91
+ | AffectNet-7 accuracy | **0.632** | 0.587 | 0.458 |
92
+ | AffectNet-7 macro-F1 | **0.632** | 0.587 | 0.410 |
93
+ | RAF-DB-7 accuracy | **0.880** | 0.673 | 0.746 |
94
+ | RAF-DB-7 macro-F1 | **0.815** | 0.586 | 0.580 |
95
+
96
+ ### AU robustness — perturbed DISFA+ (Cheong 2023 protocol)
97
+
98
+ Same 57,150 aligned DISFA+ crops as the main AU benchmark, with black-bar
99
+ occlusion and luminance shifts. The 8-AU column uses the cross-tool common set
100
+ (AU01/02/04/06/09/12/25/26).
101
+
102
+ | Perturbation | 12-AU macro-F1 | common-8 macro-F1 |
103
+ |---|---:|---:|
104
+ | none (baseline) | 0.685 | 0.774 |
105
+ | eyes occluded | 0.542 | 0.667 |
106
+ | mouth occluded | 0.538 | 0.632 |
107
+ | nose occluded | 0.678 | 0.785 |
108
+ | brightened | 0.593 | 0.684 |
109
+ | darkened | 0.675 | 0.765 |
110
+
111
+ Mouth occlusion is the worst case (−0.147 on 12-AU vs baseline).
112
+
113
+ ## Known limitations
114
+
115
+ - **AU20 is effectively non-functional** (per-AU F1 **0.057** against a 0.682
116
+ macro). AU15 (0.479) and AU06 (0.526) are also well below the macro average.
117
+ Do not rely on lip-stretch (AU20) predictions; treat AU15/AU06 with caution.
118
+ - **Gaze pitch is uneven across domains.** Per-axis pitch MAE is 3.06° on
119
+ ETH-XGaze and 7.05° on EYEDIAP, but 5.21° on MPIIGaze with a lower
120
+ prediction/ground-truth correlation (r = 0.755) — frontal, screen-directed
121
+ gaze has the narrowest pitch range and is the weakest case. Yaw is
122
+ consistently stronger than pitch on all three. Validate before depending on
123
+ absolute gaze pitch in a frontal-camera setting.
124
+ - **Landmarks are a secondary output.** The 68 points are sampled from the 478
125
+ mesh rather than predicted by a dedicated landmark head, and 300W NME is
126
+ correspondingly weaker than tools with a native 68-point head.
127
+ - Trained on posed and in-the-wild adult face imagery; performance on children,
128
+ heavy occlusion, or extreme pose is not characterized.
129
 
130
  ## Usage
131
 
 
135
  fex = detector.detect("image.jpg") # returns a py-feat Fex
136
  ```
137
 
138
+ To pin this specific checkpoint:
139
+
140
+ ```python
141
+ detector = Detectorv2(device="cuda",
142
+ multitask_weights="face_multitask_v28.safetensors")
143
+ ```
144
+
145
  The model expects a face crop produced by RetinaFace + py-feat's
146
  `extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2)`,
147
+ then center-cropped to 224 and ImageNet-normalized. `Detectorv2` handles this.
148
 
149
  ## License
150
 
151
  **Research / non-commercial use only.** Trained on datasets (AffectNet, DISFA+,
152
+ RAF-DB, Aff-Wild2, AFEW-VA, BP4D, ETH-XGaze, EYEDIAP, etc.) whose licenses
153
+ restrict use to academic research. The ConvNeXt-V2 backbone is MIT-licensed.
154
+ Confirm each constituent dataset's terms before any non-research use.