README.md

---
license: mit
tags:
  - facial-reaction-generation
  - dyadic-interaction
  - diffusion-model
  - affective-computing
  - react2026
  - mafrg
language:
  - en
pipeline_tag: other
---

# CaReDiff — Causal Reaction Diffusion (REACT 2026)


<img src="https://cdn-uploads.huggingface.co/production/uploads/65fdba70f485b5180897c642/q5J2ScpexbMFlRiEllA4p.png" alt="react_challenge_figure" width="600">

Model checkpoints for **CaReDiff**, a submission to the [REACT 2026 Challenge](https://sites.google.com/view/react2026/home) (ACM MM 2026, Multiple Appropriate Facial Reaction Generation).

Given speaker behaviour (audio, video, 3DMM coefficients, facial attributes), CaReDiff generates multiple appropriate listener facial reactions (25-d: 15 AUs + valence/arousal + 8 expressions) with an auxiliary EEG prediction head.

- **Code:** https://github.com/smu-ivpl/CaReDiff
- **Checkpoints (this repo):** https://huggingface.co/IVPL/CaReDiff

## Tracks

| Track | Architecture | Description |
|-------|-------------|-------------|
| Generic Online | PerFRDiff + EEG | Diffusion-based generation over autoregressive windows |
| Generic Offline | PerFRDiff + EEG | Diffusion-based full-sequence generation |
| Personalised Online | PerFRDiff + PRA + EEG | Frozen generic backbone + Personalised Residual Adapter (autoregressive windows) |
| Personalised Offline | PerFRDiff + PRA + EEG | Frozen generic backbone + Personalised Residual Adapter (full-sequence) |

## Repository Layout

```
CaReDiff/
├── generic/
│   ├── online/          prior + denoiser + EEG head (checkpoint_120.pth)
│   └── offline/         prior + denoiser + EEG head (checkpoint_120.pth)
└── personalised/
    ├── online/          shared backbone + 3 adapters (personality / lhfb / both)
    └── offline/         shared backbone + 3 adapters (personality / lhfb / both)
```

Each generic track folder contains:

```
<track>/
├── DiffusionPriorNetwork/checkpoint_120.pth
├── CausalTransformerDenoiser/checkpoint_120.pth
└── EEGPredictionHead/checkpoint_120.pth
```

## Usage

1. Clone the code repository:
   ```bash
   git clone https://github.com/smu-ivpl/CaReDiff
   ```
2. Set up the environment (see the [code README](https://github.com/smu-ivpl/CaReDiff)):
   ```bash
   conda create -n react python=3.10 && conda activate react
   conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia
   conda install -c fvcore -c iopath -c conda-forge fvcore iopath
   pip install -r generic/code/requirements.txt
   ```
3. Download the checkpoints from this repo and place them under `save/`. For the generic tracks:
   ```
   save/motion_diffusion/react_2025/
   ├── online/checkpoints/pretrained/
   │   ├── DiffusionPriorNetwork/checkpoint_120.pth
   │   ├── CausalTransformerDenoiser/checkpoint_120.pth
   │   └── EEGPredictionHead/checkpoint_120.pth
   └── offline/checkpoints/pretrained/
       └── ... (same structure)
   ```
4. Run evaluation (see below).

Full placement instructions, SHA-256 checksums, and per-track details are in each variant's `checkpoints/README.md` in the code repository.

### Generic Evaluation

```bash
# Generic Online
python main.py --config-name generic_online/motion_diffusion \
    stage=test data_dir=./datasets/REACT2026/ \
    trainer.batch_size=1 resume_id=pretrained

# Generic Offline
python main.py --config-name generic_offline/motion_diffusion \
    stage=test data_dir=./datasets/REACT2026/ \
    trainer.batch_size=1 resume_id=pretrained
```

### Personalised Evaluation

Each personalised track uses a frozen generic backbone plus one of three
condition adapters (`personality` / `lhfb` / `both`). Download the
`personalised/<track>/` folder and set `PKG` to its parent absolute path; the
commands reference every weight explicitly through `PKG`.

```bash
PKG=<absolute path of the downloaded personalised folder>

# Personalised Offline (personality condition)
python main.py --config-name g2p_delta stage=test task=offline \
  data_dir=<MARS_ROOT> run_id=eval_offline_personality \
  trainer.batch_size=4 num_gts=10 \
  trainer.generic.eval_condition_mode=matched \
  trainer.generic.eval_eeg=false \
  trainer.main_model.args.personal_condition_mode=personality_only \
  resume_id=personality \
  trainer.ckpt_dir=$PKG/offline/adapters \
  trainer.pretrained.diffusion_decoder=$PKG/offline/backbone/CausalTransformerDenoiser/checkpoint_120.pth \
  trainer.pretrained.diffusion_prior=$PKG/offline/backbone/DiffusionPriorNetwork/checkpoint_120.pth \
  trainer.pretrained.eeg_head_checkpoint=$PKG/offline/backbone/EEGPredictionHead/checkpoint_120.pth

# Personalised Online (personality condition)
python main.py --config-name g2p_delta_online stage=test task=online \
  data_dir=<MARS_ROOT> run_id=eval_online_personality \
  trainer.batch_size=4 num_gts=10 \
  trainer.generic.eval_condition_mode=matched \
  trainer.generic.eval_eeg=false \
  trainer.main_model.args.personal_condition_mode=personality_only \
  trainer.model.diff_model.diffusion_decoder.args.past_l_emotion_drop_prob=0.2 \
  resume_id=personality \
  trainer.ckpt_dir=$PKG/online/adapters \
  trainer.pretrained.diffusion_decoder=$PKG/online/backbone/CausalTransformerDenoiser/checkpoint_120.pth \
  trainer.pretrained.diffusion_prior=$PKG/online/backbone/DiffusionPriorNetwork/checkpoint_120.pth \
  trainer.pretrained.eeg_head_checkpoint=$PKG/online/backbone/EEGPredictionHead/checkpoint_120.pth
```

For the `lhfb` / `both` conditions, change `resume_id` (`lhfb` or `both`) and
`personal_condition_mode` (`3dmm_only` or `3dmm_personality`) accordingly. For
the online track, keep `past_l_emotion_drop_prob=0.2` — it enables the
past-listener conditioning pathway and is required to reproduce the reported
numbers.

## Additional Requirements

- **Post-processor (EmotionVAE) checkpoint** — required for evaluation on every `stage=test` run. Obtain from the [official REACT 2026 baseline](https://github.com/reactmultimodalchallenge/baseline_react2026) and place at `pretrained_models/post_processor/checkpoint.pth`.
- **PIRender + FaceVerse** — needed only for FRRea (rendered-frame FID) evaluation.
- **MARS dataset** — obtain through the challenge organisers.

## Metrics

| Metric | Description |
|--------|-------------|
| FRCorr ↑ | Facial Reaction Correlation (CCC against GT) |
| FRDist ↓ | Facial Reaction Distance (DTW against GT) |
| FRDiv ↑ | Diversity across the 10 generated predictions (pairwise MSE) |
| FRVar ↑ | Temporal variance within a generated reaction |
| FRRea ↓ | Realism (FID on rendered frames) |
| FRSyn ↓ | Synchrony (Time-Lagged Cross-Correlation) |

## Citation

```bibtex
@article{song2023multiple,
  title={Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?},
  author={Song, Siyang and Spitale, Micol and Luo, Yiming and Bal, Batuhan and Gunes, Hatice},
  journal={arXiv preprint arXiv:2302.06514},
  year={2023}
}

@inproceedings{song2025react,
  title={React 2025: the third multiple appropriate facial reaction generation challenge},
  author={Song, Siyang and Spitale, Micol and Kong, Xiangyu and Zhu, Hengde and Luo, Cheng and Palmero, Cristina and Barquero, German and others},
  booktitle={Proceedings of the 33rd ACM International Conference on Multimedia},
  pages={13979--13984},
  year={2025}
}
```

## Acknowledgement

- [FaceVerse](https://github.com/LizhenWangT/FaceVerse)
- [PIRender](https://github.com/RenYurui/PIRender)
- [REACT 2026 Baseline](https://github.com/reactmultimodalchallenge/baseline_react2026)

## License

MIT