File size: 4,751 Bytes
b9ce39d 56aaaac b9ce39d 56aaaac b9ce39d 56aaaac b9ce39d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
license: mit
tags:
- facial-expression-recognition
- action-units
- valence-arousal
- affective-computing
- abaw
- rectified-flow
- dinov3
pipeline_tag: image-classification
---
# AffectFlow-DINO
Uncertainty-aware multi-task facial affect estimation for the **11th ABAW Multi-Task Learning
challenge**: DINOv3 backbone + deterministic VA/expression/AU heads + a conditional
rectified-flow head over the joint 22-dimensional affect vector.
Code, training scripts, and the full 28-experiment ablation study:
**[github.com/Bekhouche/AffectFlow-DINO](https://github.com/Bekhouche/AffectFlow-DINO)**.
**Paper:** [AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional
Rectified Flow](https://arxiv.org/abs/2607.13250) — Salah Eddine Bekhouche, Abdellah Zakaria
Sellam, Fadi Dornaika, Abdenour Hadid. arXiv:2607.13250, 2026.
## Quickstart
```bash
pip install torch torchvision transformers huggingface_hub pillow
git clone https://github.com/Bekhouche/AffectFlow-DINO.git && cd AffectFlow-DINO
python inference.py --model finetune-flow-retune-b10 --image face.jpg
```
## Models in this repo
Each subfolder is one checkpoint: `model.pt` (weights + architecture config) plus, where
applicable, `au_thresholds.json` and/or `expr_weights.json` post-hoc calibration files that
`inference.py` applies automatically.
| Folder | Backbone | Best decode | P_MTL (calibrated) | P_VA | P_EXPR | P_AU |
|---|---|---|---|---|---|---|
| [`finetune-flow-retune-b10`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/finetune-flow-retune-b10) | ViT-S/16, fine-tuned | Det | **1.177** | 0.325 | 0.350 | 0.502 |
| [`vitb-finetune`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/vitb-finetune) | ViT-B/16, fine-tuned | Det | 1.116 | 0.354 | 0.255 | 0.507 |
| [`finetune-flow-retune-b05`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/finetune-flow-retune-b05) | ViT-S/16, fine-tuned | Det / Flow | 1.062 / 0.956 | 0.291 | 0.303 | 0.469 |
| [`finetune`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/finetune) | ViT-S/16, fine-tuned | Det | 1.101 | 0.318 | 0.285 | 0.497 |
| [`au-posw-cw`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/au-posw-cw) | ViT-S/16, frozen | Det | 0.890 (no cal.) | 0.211 | 0.240 | 0.439 |
| [`psp`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/psp) | ViT-S/16, frozen (patch soft-pool) | Det | 0.859 (no cal.) | 0.235 | 0.236 | 0.389 |
| [`affectflow-base`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/affectflow-base) | ViT-S/16, frozen | Flow | 0.888 | 0.237 | 0.210 | 0.441 |
| [`det-baseline`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/det-baseline) | ViT-S/16, frozen | Det | 0.793 (no cal.) | 0.199 | 0.216 | 0.378 |
`P_MTL = P_VA + P_EXPR + P_AU` is the official ABAW MTL composite metric (P_VA = mean CCC of
valence/arousal, P_EXPR = expression macro-F1, P_AU = mean AU F1), evaluated on the s-Aff-Wild2
validation split. Official challenge baseline: P_MTL = 0.450.
These 8 checkpoints are a curated subset of the ~30 ablation runs behind the paper — one per
architecturally distinct configuration. The full ablation table, including sweeps that reused
one of these architectures with different loss weights/regularization, is in the GitHub repo's
[EXPERIMENTS.md](https://github.com/Bekhouche/AffectFlow-DINO/blob/main/EXPERIMENTS.md).
## Input / output
- Input: a cropped, roughly frontal face image, resized to 224x224 and ImageNet-normalized
(handled automatically by `inference.py`).
- Output: valence, arousal in `[-1, 1]`; one of 8 expressions (Neutral, Anger, Disgust, Fear,
Happiness, Sadness, Surprise, Other); a subset of 12 active Action Units (AU1, AU2, AU4, AU6,
AU7, AU10, AU12, AU15, AU23, AU24, AU25, AU26).
- Two decode modes: `deterministic` (task heads directly) or `flow` (average of N sampled
rectified-flow trajectories — enables uncertainty estimation via trajectory spread).
## Limitations
Trained and validated only on s-Aff-Wild2 (in-the-wild but frame-level, no temporal context).
Expression prediction remains the weakest task (P_EXPR <= 0.35): Fear and Sadness are rare
classes that only partially recover under calibration. Not evaluated on demographic subgroups;
treat outputs as research signals, not clinical or high-stakes decisions.
## Citation
```bibtex
@article{bekhouche2026affectflowdino,
title = {AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow},
author = {Bekhouche, Salah Eddine and Sellam, Abdellah Zakaria and Dornaika, Fadi and Hadid, Abdenour},
journal = {arXiv preprint arXiv:2607.13250},
year = {2026}
}
```
## License
MIT.
|