Hugging Face's logo

IVPL
/
CaReDiff

Other
English
facial-reaction-generation
dyadic-interaction
diffusion-model
affective-computing
react2026
mafrg
Model card Files
xet
 Community
CaReDiff / personalised /offline /README.md
hanseeul's picture
hanseeul
Point READMEs to the GitHub repository for source code
5d0bdd4 verified
preview code
|
Raw
History Blame
4.5 kB

CaReDiff Personalised Models, Offline Track (REACT 2026)

Three personalised models for the offline MAFRG track. Each model is the same frozen generic offline backbone plus a Personalised Residual Adapter (PRA) trained under a different listener condition. The backbone weights are shared by all three and are identical to the generic offline submission.

Layout 

offline/
  backbone/                 frozen generic backbone (shared by all conditions)
    CausalTransformerDenoiser/checkpoint_120.pth
    DiffusionPriorNetwork/checkpoint_120.pth
    EEGPredictionHead/checkpoint_120.pth
  adapters/
    personality/ModifierNetwork/checkpoint_best.pth
    lhfb/ModifierNetwork/checkpoint_best.pth
    both/ModifierNetwork/checkpoint_best.pth

The adapter file also contains the fine-tuned EEG head, which overwrites the backbone EEG head at load time.

Checksums (SHA-256)

File SHA-256
backbone/CausalTransformerDenoiser/checkpoint_120.pth 68faca9700415c949eecbe7bd3e381877a76b5e1b24bdab9c30e6fd5b628faa2
backbone/DiffusionPriorNetwork/checkpoint_120.pth d1b66e87f51afd9bb93bdcef1b9e350e6366aa8f995920e400d7e7dd4e299357
backbone/EEGPredictionHead/checkpoint_120.pth 750c49999a180cda330b88d771f99d1dca0fd94a810470ea77a45561cfd58780
adapters/personality/ModifierNetwork/checkpoint_best.pth 8e0a501237c9b80b8c9e9524bd089fa5ca54ad747bdf9ed65dd97b8d883bf928
adapters/lhfb/ModifierNetwork/checkpoint_best.pth 0ddfde5284c580c3cc461006b2b7cd4df73d2715a8700d3838d8d5e5db8eb7f4
adapters/both/ModifierNetwork/checkpoint_best.pth 73434669c633bc6384acc9845e62c0c4302c9322be27f04e30005e55dda3ab92

Conditions

Folder Listener condition Config value
adapters/personality Big-Five personality (5-d) personality_only
adapters/lhfb Listener historical facial behaviour (3DMM) 3dmm_only
adapters/both Both, gated fusion 3dmm_personality

Training: AdamW, learning rate 2e-4, weight decay 1e-4, gradient clipping 1.0, 30 epochs, batch size 32, seed 1234, counterfactual listener-swap loss (weight 0.5, margin 0.05). The backbone stays frozen throughout.

Test performance (MARS test set, official evaluation code, num_gts=10)

Condition FRCorr FRDist FRDiv FRVar FRRea FRSyn
personality 0.7786 173.63 0.1221 0.0782 50.91 48.37
lhfb 0.7824 173.11 0.1200 0.0766 51.23 48.26
both 0.7822 171.41 0.1187 0.0761 50.82 48.28

FRRea is the FID between rendered generated frames and ground-truth frames (56,100 frames per side, frame stride 30).

How to run

The source code is in the CaReDiff GitHub repository (https://github.com/smu-ivpl/CaReDiff, personalised/code/). Example for the personality condition (set PKG to the absolute path of the personalised folder containing the checkpoints):

cd code
python main.py --config-name g2p_delta stage=test task=offline \
  data_dir=<MARS_ROOT> run_id=eval_offline_personality \
  trainer.batch_size=4 num_gts=10 \
  trainer.generic.eval_condition_mode=matched \
  trainer.generic.eval_eeg=false \
  trainer.main_model.args.personal_condition_mode=personality_only \
  resume_id=personality \
  trainer.ckpt_dir=$PKG/offline/adapters \
  trainer.pretrained.diffusion_decoder=$PKG/offline/backbone/CausalTransformerDenoiser/checkpoint_120.pth \
  trainer.pretrained.diffusion_prior=$PKG/offline/backbone/DiffusionPriorNetwork/checkpoint_120.pth \
  trainer.pretrained.eeg_head_checkpoint=$PKG/offline/backbone/EEGPredictionHead/checkpoint_120.pth

The adapter is loaded from <trainer.ckpt_dir>/<resume_id>/ModifierNetwork/, which maps directly onto the adapters/ layout above. For the other two conditions, change personal_condition_mode and resume_id (lhfb or both) according to the table. The loader verifies that the checkpoint was trained with the configured condition mode and stops with an error on a mismatch.

Notes

  • Large assets shared with the official baseline are not duplicated here. The post-processor EmotionVAE checkpoint (517 MB) is required for evaluation and must be placed at code/pretrained_models/post_processor/checkpoint.pth. The PIRender renderer (234 MB) is needed only for FRRea rendering. Take both from the official baseline_react2026 repository.
  • Python dependencies: code/requirements.txt.
  • The MARS dataset is not included and must be obtained through the challenge organisers.