CaReDiff — Causal Reaction Diffusion (REACT 2026)

Model checkpoints for CaReDiff, a submission to the REACT 2026 Challenge (ACM MM 2026, Multiple Appropriate Facial Reaction Generation).

Given speaker behaviour (audio, video, 3DMM coefficients, facial attributes), CaReDiff generates multiple appropriate listener facial reactions (25-d: 15 AUs + valence/arousal + 8 expressions) with an auxiliary EEG prediction head.

Code: https://github.com/smu-ivpl/CaReDiff
Checkpoints (this repo): https://huggingface.co/IVPL/CaReDiff

Tracks

Track	Architecture	Description
Generic Online	PerFRDiff + EEG	Diffusion-based generation over autoregressive windows
Generic Offline	PerFRDiff + EEG	Diffusion-based full-sequence generation
Personalised Online	PerFRDiff + PRA + EEG	Frozen generic backbone + Personalised Residual Adapter (autoregressive windows)
Personalised Offline	PerFRDiff + PRA + EEG	Frozen generic backbone + Personalised Residual Adapter (full-sequence)

Repository Layout

CaReDiff/
├── generic/
│   ├── online/          prior + denoiser + EEG head (checkpoint_120.pth)
│   └── offline/         prior + denoiser + EEG head (checkpoint_120.pth)
└── personalised/
    ├── online/          shared backbone + 3 adapters (personality / lhfb / both)
    └── offline/         shared backbone + 3 adapters (personality / lhfb / both)

Each generic track folder contains:

<track>/
├── DiffusionPriorNetwork/checkpoint_120.pth
├── CausalTransformerDenoiser/checkpoint_120.pth
└── EEGPredictionHead/checkpoint_120.pth

Usage

Clone the code repository:

git clone https://github.com/smu-ivpl/CaReDiff

Set up the environment (see the code README):

conda create -n react python=3.10 && conda activate react
conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
pip install -r generic/code/requirements.txt

Download the checkpoints from this repo and place them under save/. For the generic tracks:

save/motion_diffusion/react_2025/
├── online/checkpoints/pretrained/
│   ├── DiffusionPriorNetwork/checkpoint_120.pth
│   ├── CausalTransformerDenoiser/checkpoint_120.pth
│   └── EEGPredictionHead/checkpoint_120.pth
└── offline/checkpoints/pretrained/
    └── ... (same structure)

Run evaluation (see below).

Full placement instructions, SHA-256 checksums, and per-track details are in each variant's checkpoints/README.md in the code repository.

Generic Evaluation

# Generic Online
python main.py --config-name generic_online/motion_diffusion \
    stage=test data_dir=./datasets/REACT2026/ \
    trainer.batch_size=1 resume_id=pretrained

# Generic Offline
python main.py --config-name generic_offline/motion_diffusion \
    stage=test data_dir=./datasets/REACT2026/ \
    trainer.batch_size=1 resume_id=pretrained

Personalised Evaluation

Each personalised track uses a frozen generic backbone plus one of three condition adapters (personality / lhfb / both). Download the personalised/<track>/ folder and set PKG to its parent absolute path; the commands reference every weight explicitly through PKG.

PKG=<absolute path of the downloaded personalised folder>

# Personalised Offline (personality condition)
python main.py --config-name g2p_delta stage=test task=offline \
  data_dir=<MARS_ROOT> run_id=eval_offline_personality \
  trainer.batch_size=4 num_gts=10 \
  trainer.generic.eval_condition_mode=matched \
  trainer.generic.eval_eeg=false \
  trainer.main_model.args.personal_condition_mode=personality_only \
  resume_id=personality \
  trainer.ckpt_dir=$PKG/offline/adapters \
  trainer.pretrained.diffusion_decoder=$PKG/offline/backbone/CausalTransformerDenoiser/checkpoint_120.pth \
  trainer.pretrained.diffusion_prior=$PKG/offline/backbone/DiffusionPriorNetwork/checkpoint_120.pth \
  trainer.pretrained.eeg_head_checkpoint=$PKG/offline/backbone/EEGPredictionHead/checkpoint_120.pth

# Personalised Online (personality condition)
python main.py --config-name g2p_delta_online stage=test task=online \
  data_dir=<MARS_ROOT> run_id=eval_online_personality \
  trainer.batch_size=4 num_gts=10 \
  trainer.generic.eval_condition_mode=matched \
  trainer.generic.eval_eeg=false \
  trainer.main_model.args.personal_condition_mode=personality_only \
  trainer.model.diff_model.diffusion_decoder.args.past_l_emotion_drop_prob=0.2 \
  resume_id=personality \
  trainer.ckpt_dir=$PKG/online/adapters \
  trainer.pretrained.diffusion_decoder=$PKG/online/backbone/CausalTransformerDenoiser/checkpoint_120.pth \
  trainer.pretrained.diffusion_prior=$PKG/online/backbone/DiffusionPriorNetwork/checkpoint_120.pth \
  trainer.pretrained.eeg_head_checkpoint=$PKG/online/backbone/EEGPredictionHead/checkpoint_120.pth

For the lhfb / both conditions, change resume_id (lhfb or both) and personal_condition_mode (3dmm_only or 3dmm_personality) accordingly. For the online track, keep past_l_emotion_drop_prob=0.2 — it enables the past-listener conditioning pathway and is required to reproduce the reported numbers.

Additional Requirements

Post-processor (EmotionVAE) checkpoint — required for evaluation on every stage=test run. Obtain from the official REACT 2026 baseline and place at pretrained_models/post_processor/checkpoint.pth.
PIRender + FaceVerse — needed only for FRRea (rendered-frame FID) evaluation.
MARS dataset — obtain through the challenge organisers.

Metrics

Metric	Description
FRCorr ↑	Facial Reaction Correlation (CCC against GT)
FRDist ↓	Facial Reaction Distance (DTW against GT)
FRDiv ↑	Diversity across the 10 generated predictions (pairwise MSE)
FRVar ↑	Temporal variance within a generated reaction
FRRea ↓	Realism (FID on rendered frames)
FRSyn ↓	Synchrony (Time-Lagged Cross-Correlation)

Citation

@article{song2023multiple,
  title={Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?},
  author={Song, Siyang and Spitale, Micol and Luo, Yiming and Bal, Batuhan and Gunes, Hatice},
  journal={arXiv preprint arXiv:2302.06514},
  year={2023}
}

@inproceedings{song2025react,
  title={React 2025: the third multiple appropriate facial reaction generation challenge},
  author={Song, Siyang and Spitale, Micol and Kong, Xiangyu and Zhu, Hengde and Luo, Cheng and Palmero, Cristina and Barquero, German and others},
  booktitle={Proceedings of the 33rd ACM International Conference on Multimedia},
  pages={13979--13984},
  year={2025}
}

Acknowledgement

License

MIT

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

Other

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for IVPL/CaReDiff

Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?

Paper • 2302.06514 • Published Mar 23, 2023