README.md

---
license: mit
tags:
  - facial-expression-recognition
  - action-units
  - valence-arousal
  - affective-computing
  - abaw
  - rectified-flow
  - dinov3
pipeline_tag: image-classification
---

# AffectFlow-DINO

Uncertainty-aware multi-task facial affect estimation for the **11th ABAW Multi-Task Learning
challenge**: DINOv3 backbone + deterministic VA/expression/AU heads + a conditional
rectified-flow head over the joint 22-dimensional affect vector.

Code, training scripts, and the full 28-experiment ablation study:
**[github.com/Bekhouche/AffectFlow-DINO](https://github.com/Bekhouche/AffectFlow-DINO)**.

**Paper:** [AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional
Rectified Flow](https://arxiv.org/abs/2607.13250) — Salah Eddine Bekhouche, Abdellah Zakaria
Sellam, Fadi Dornaika, Abdenour Hadid. arXiv:2607.13250, 2026.

## Quickstart

```bash
pip install torch torchvision transformers huggingface_hub pillow
git clone https://github.com/Bekhouche/AffectFlow-DINO.git && cd AffectFlow-DINO
python inference.py --model finetune-flow-retune-b10 --image face.jpg
```

## Models in this repo

Each subfolder is one checkpoint: `model.pt` (weights + architecture config) plus, where
applicable, `au_thresholds.json` and/or `expr_weights.json` post-hoc calibration files that
`inference.py` applies automatically.

| Folder | Backbone | Best decode | P_MTL (calibrated) | P_VA | P_EXPR | P_AU |
|---|---|---|---|---|---|---|
| [`finetune-flow-retune-b10`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/finetune-flow-retune-b10) | ViT-S/16, fine-tuned | Det | **1.177** | 0.325 | 0.350 | 0.502 |
| [`vitb-finetune`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/vitb-finetune) | ViT-B/16, fine-tuned | Det | 1.116 | 0.354 | 0.255 | 0.507 |
| [`finetune-flow-retune-b05`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/finetune-flow-retune-b05) | ViT-S/16, fine-tuned | Det / Flow | 1.062 / 0.956 | 0.291 | 0.303 | 0.469 |
| [`finetune`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/finetune) | ViT-S/16, fine-tuned | Det | 1.101 | 0.318 | 0.285 | 0.497 |
| [`au-posw-cw`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/au-posw-cw) | ViT-S/16, frozen | Det | 0.890 (no cal.) | 0.211 | 0.240 | 0.439 |
| [`psp`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/psp) | ViT-S/16, frozen (patch soft-pool) | Det | 0.859 (no cal.) | 0.235 | 0.236 | 0.389 |
| [`affectflow-base`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/affectflow-base) | ViT-S/16, frozen | Flow | 0.888 | 0.237 | 0.210 | 0.441 |
| [`det-baseline`](https://huggingface.co/Bekhouche/AffectFlow-DINO/tree/main/det-baseline) | ViT-S/16, frozen | Det | 0.793 (no cal.) | 0.199 | 0.216 | 0.378 |

`P_MTL = P_VA + P_EXPR + P_AU` is the official ABAW MTL composite metric (P_VA = mean CCC of
valence/arousal, P_EXPR = expression macro-F1, P_AU = mean AU F1), evaluated on the s-Aff-Wild2
validation split. Official challenge baseline: P_MTL = 0.450.

These 8 checkpoints are a curated subset of the ~30 ablation runs behind the paper — one per
architecturally distinct configuration. The full ablation table, including sweeps that reused
one of these architectures with different loss weights/regularization, is in the GitHub repo's
[EXPERIMENTS.md](https://github.com/Bekhouche/AffectFlow-DINO/blob/main/EXPERIMENTS.md).

## Input / output

- Input: a cropped, roughly frontal face image, resized to 224x224 and ImageNet-normalized
  (handled automatically by `inference.py`).
- Output: valence, arousal in `[-1, 1]`; one of 8 expressions (Neutral, Anger, Disgust, Fear,
  Happiness, Sadness, Surprise, Other); a subset of 12 active Action Units (AU1, AU2, AU4, AU6,
  AU7, AU10, AU12, AU15, AU23, AU24, AU25, AU26).
- Two decode modes: `deterministic` (task heads directly) or `flow` (average of N sampled
  rectified-flow trajectories — enables uncertainty estimation via trajectory spread).

## Limitations

Trained and validated only on s-Aff-Wild2 (in-the-wild but frame-level, no temporal context).
Expression prediction remains the weakest task (P_EXPR <= 0.35): Fear and Sadness are rare
classes that only partially recover under calibration. Not evaluated on demographic subgroups;
treat outputs as research signals, not clinical or high-stakes decisions.

## Citation

```bibtex
@article{bekhouche2026affectflowdino,
  title         = {AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow},
  author        = {Bekhouche, Salah Eddine and Sellam, Abdellah Zakaria and Dornaika, Fadi and Hadid, Abdenour},
  journal       = {arXiv preprint arXiv:2607.13250},
  year          = {2026}
}
```

## License

MIT.