--- license: mit tags: - facial-reaction-generation - dyadic-interaction - diffusion-model - affective-computing - react2026 - mafrg language: - en pipeline_tag: other --- # CaReDiff — Causal Reaction Diffusion (REACT 2026) react_challenge_figure Model checkpoints for **CaReDiff**, a submission to the [REACT 2026 Challenge](https://sites.google.com/view/react2026/home) (ACM MM 2026, Multiple Appropriate Facial Reaction Generation). Given speaker behaviour (audio, video, 3DMM coefficients, facial attributes), CaReDiff generates multiple appropriate listener facial reactions (25-d: 15 AUs + valence/arousal + 8 expressions) with an auxiliary EEG prediction head. - **Code:** https://github.com/smu-ivpl/CaReDiff - **Checkpoints (this repo):** https://huggingface.co/IVPL/CaReDiff ## Tracks | Track | Architecture | Description | |-------|-------------|-------------| | Generic Online | PerFRDiff + EEG | Diffusion-based generation over autoregressive windows | | Generic Offline | PerFRDiff + EEG | Diffusion-based full-sequence generation | | Personalised Online | PerFRDiff + PRA + EEG | Frozen generic backbone + Personalised Residual Adapter (autoregressive windows) | | Personalised Offline | PerFRDiff + PRA + EEG | Frozen generic backbone + Personalised Residual Adapter (full-sequence) | ## Repository Layout ``` CaReDiff/ ├── generic/ │ ├── online/ prior + denoiser + EEG head (checkpoint_120.pth) │ └── offline/ prior + denoiser + EEG head (checkpoint_120.pth) └── personalised/ ├── online/ shared backbone + 3 adapters (personality / lhfb / both) └── offline/ shared backbone + 3 adapters (personality / lhfb / both) ``` Each generic track folder contains: ``` / ├── DiffusionPriorNetwork/checkpoint_120.pth ├── CausalTransformerDenoiser/checkpoint_120.pth └── EEGPredictionHead/checkpoint_120.pth ``` ## Usage 1. Clone the code repository: ```bash git clone https://github.com/smu-ivpl/CaReDiff ``` 2. Set up the environment (see the [code README](https://github.com/smu-ivpl/CaReDiff)): ```bash conda create -n react python=3.10 && conda activate react conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia conda install -c fvcore -c iopath -c conda-forge fvcore iopath pip install -r generic/code/requirements.txt ``` 3. Download the checkpoints from this repo and place them under `save/`. For the generic tracks: ``` save/motion_diffusion/react_2025/ ├── online/checkpoints/pretrained/ │ ├── DiffusionPriorNetwork/checkpoint_120.pth │ ├── CausalTransformerDenoiser/checkpoint_120.pth │ └── EEGPredictionHead/checkpoint_120.pth └── offline/checkpoints/pretrained/ └── ... (same structure) ``` 4. Run evaluation (see below). Full placement instructions, SHA-256 checksums, and per-track details are in each variant's `checkpoints/README.md` in the code repository. ### Generic Evaluation ```bash # Generic Online python main.py --config-name generic_online/motion_diffusion \ stage=test data_dir=./datasets/REACT2026/ \ trainer.batch_size=1 resume_id=pretrained # Generic Offline python main.py --config-name generic_offline/motion_diffusion \ stage=test data_dir=./datasets/REACT2026/ \ trainer.batch_size=1 resume_id=pretrained ``` ### Personalised Evaluation Each personalised track uses a frozen generic backbone plus one of three condition adapters (`personality` / `lhfb` / `both`). Download the `personalised//` folder and set `PKG` to its parent absolute path; the commands reference every weight explicitly through `PKG`. ```bash PKG= # Personalised Offline (personality condition) python main.py --config-name g2p_delta stage=test task=offline \ data_dir= run_id=eval_offline_personality \ trainer.batch_size=4 num_gts=10 \ trainer.generic.eval_condition_mode=matched \ trainer.generic.eval_eeg=false \ trainer.main_model.args.personal_condition_mode=personality_only \ resume_id=personality \ trainer.ckpt_dir=$PKG/offline/adapters \ trainer.pretrained.diffusion_decoder=$PKG/offline/backbone/CausalTransformerDenoiser/checkpoint_120.pth \ trainer.pretrained.diffusion_prior=$PKG/offline/backbone/DiffusionPriorNetwork/checkpoint_120.pth \ trainer.pretrained.eeg_head_checkpoint=$PKG/offline/backbone/EEGPredictionHead/checkpoint_120.pth # Personalised Online (personality condition) python main.py --config-name g2p_delta_online stage=test task=online \ data_dir= run_id=eval_online_personality \ trainer.batch_size=4 num_gts=10 \ trainer.generic.eval_condition_mode=matched \ trainer.generic.eval_eeg=false \ trainer.main_model.args.personal_condition_mode=personality_only \ trainer.model.diff_model.diffusion_decoder.args.past_l_emotion_drop_prob=0.2 \ resume_id=personality \ trainer.ckpt_dir=$PKG/online/adapters \ trainer.pretrained.diffusion_decoder=$PKG/online/backbone/CausalTransformerDenoiser/checkpoint_120.pth \ trainer.pretrained.diffusion_prior=$PKG/online/backbone/DiffusionPriorNetwork/checkpoint_120.pth \ trainer.pretrained.eeg_head_checkpoint=$PKG/online/backbone/EEGPredictionHead/checkpoint_120.pth ``` For the `lhfb` / `both` conditions, change `resume_id` (`lhfb` or `both`) and `personal_condition_mode` (`3dmm_only` or `3dmm_personality`) accordingly. For the online track, keep `past_l_emotion_drop_prob=0.2` — it enables the past-listener conditioning pathway and is required to reproduce the reported numbers. ## Additional Requirements - **Post-processor (EmotionVAE) checkpoint** — required for evaluation on every `stage=test` run. Obtain from the [official REACT 2026 baseline](https://github.com/reactmultimodalchallenge/baseline_react2026) and place at `pretrained_models/post_processor/checkpoint.pth`. - **PIRender + FaceVerse** — needed only for FRRea (rendered-frame FID) evaluation. - **MARS dataset** — obtain through the challenge organisers. ## Metrics | Metric | Description | |--------|-------------| | FRCorr ↑ | Facial Reaction Correlation (CCC against GT) | | FRDist ↓ | Facial Reaction Distance (DTW against GT) | | FRDiv ↑ | Diversity across the 10 generated predictions (pairwise MSE) | | FRVar ↑ | Temporal variance within a generated reaction | | FRRea ↓ | Realism (FID on rendered frames) | | FRSyn ↓ | Synchrony (Time-Lagged Cross-Correlation) | ## Citation ```bibtex @article{song2023multiple, title={Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?}, author={Song, Siyang and Spitale, Micol and Luo, Yiming and Bal, Batuhan and Gunes, Hatice}, journal={arXiv preprint arXiv:2302.06514}, year={2023} } @inproceedings{song2025react, title={React 2025: the third multiple appropriate facial reaction generation challenge}, author={Song, Siyang and Spitale, Micol and Kong, Xiangyu and Zhu, Hengde and Luo, Cheng and Palmero, Cristina and Barquero, German and others}, booktitle={Proceedings of the 33rd ACM International Conference on Multimedia}, pages={13979--13984}, year={2025} } ``` ## Acknowledgement - [FaceVerse](https://github.com/LizhenWangT/FaceVerse) - [PIRender](https://github.com/RenYurui/PIRender) - [REACT 2026 Baseline](https://github.com/reactmultimodalchallenge/baseline_react2026) ## License MIT