personalised/offline/README.md

# CaReDiff Personalised Models, Offline Track (REACT 2026)

Three personalised models for the offline MAFRG track. Each model is the same
frozen generic offline backbone plus a Personalised Residual Adapter (PRA)
trained under a different listener condition. The backbone weights are shared
by all three and are identical to the generic offline submission.

## Layout

```
offline/
  backbone/                 frozen generic backbone (shared by all conditions)
    CausalTransformerDenoiser/checkpoint_120.pth
    DiffusionPriorNetwork/checkpoint_120.pth
    EEGPredictionHead/checkpoint_120.pth
  adapters/
    personality/ModifierNetwork/checkpoint_best.pth
    lhfb/ModifierNetwork/checkpoint_best.pth
    both/ModifierNetwork/checkpoint_best.pth
```

The adapter file also contains the fine-tuned EEG head, which overwrites the
backbone EEG head at load time.

## Checksums (SHA-256)

| File | SHA-256 |
|---|---|
| backbone/CausalTransformerDenoiser/checkpoint_120.pth | 68faca9700415c949eecbe7bd3e381877a76b5e1b24bdab9c30e6fd5b628faa2 |
| backbone/DiffusionPriorNetwork/checkpoint_120.pth | d1b66e87f51afd9bb93bdcef1b9e350e6366aa8f995920e400d7e7dd4e299357 |
| backbone/EEGPredictionHead/checkpoint_120.pth | 750c49999a180cda330b88d771f99d1dca0fd94a810470ea77a45561cfd58780 |
| adapters/personality/ModifierNetwork/checkpoint_best.pth | 8e0a501237c9b80b8c9e9524bd089fa5ca54ad747bdf9ed65dd97b8d883bf928 |
| adapters/lhfb/ModifierNetwork/checkpoint_best.pth | 0ddfde5284c580c3cc461006b2b7cd4df73d2715a8700d3838d8d5e5db8eb7f4 |
| adapters/both/ModifierNetwork/checkpoint_best.pth | 73434669c633bc6384acc9845e62c0c4302c9322be27f04e30005e55dda3ab92 |

## Conditions

| Folder | Listener condition | Config value |
|---|---|---|
| adapters/personality | Big-Five personality (5-d) | `personality_only` |
| adapters/lhfb | Listener historical facial behaviour (3DMM) | `3dmm_only` |
| adapters/both | Both, gated fusion | `3dmm_personality` |

Training: AdamW, learning rate 2e-4, weight decay 1e-4, gradient clipping 1.0,
30 epochs, batch size 32, seed 1234, counterfactual listener-swap loss
(weight 0.5, margin 0.05). The backbone stays frozen throughout.

## Test performance (MARS test set, official evaluation code, num_gts=10)

| Condition | FRCorr | FRDist | FRDiv | FRVar | FRRea | FRSyn |
|---|---|---|---|---|---|---|
| personality | 0.7786 | 173.63 | 0.1221 | 0.0782 | 50.91 | 48.37 |
| lhfb | 0.7824 | 173.11 | 0.1200 | 0.0766 | 51.23 | 48.26 |
| both | 0.7822 | 171.41 | 0.1187 | 0.0761 | 50.82 | 48.28 |

FRRea is the FID between rendered generated frames and ground-truth frames
(56,100 frames per side, frame stride 30).

## How to run

The source code is in the CaReDiff GitHub repository
(https://github.com/smu-ivpl/CaReDiff, `personalised/code/`). Example for
the personality condition (set `PKG` to the absolute path of the
`personalised` folder containing the checkpoints):

```bash
cd code
python main.py --config-name g2p_delta stage=test task=offline \
  data_dir=<MARS_ROOT> run_id=eval_offline_personality \
  trainer.batch_size=4 num_gts=10 \
  trainer.generic.eval_condition_mode=matched \
  trainer.generic.eval_eeg=false \
  trainer.main_model.args.personal_condition_mode=personality_only \
  resume_id=personality \
  trainer.ckpt_dir=$PKG/offline/adapters \
  trainer.pretrained.diffusion_decoder=$PKG/offline/backbone/CausalTransformerDenoiser/checkpoint_120.pth \
  trainer.pretrained.diffusion_prior=$PKG/offline/backbone/DiffusionPriorNetwork/checkpoint_120.pth \
  trainer.pretrained.eeg_head_checkpoint=$PKG/offline/backbone/EEGPredictionHead/checkpoint_120.pth
```

The adapter is loaded from `<trainer.ckpt_dir>/<resume_id>/ModifierNetwork/`,
which maps directly onto the `adapters/` layout above. For the other two
conditions, change `personal_condition_mode` and `resume_id` (`lhfb` or
`both`) according to the table. The loader verifies that the checkpoint was
trained with the configured condition mode and stops with an error on a
mismatch.

## Notes

- Large assets shared with the official baseline are not duplicated here.
  The post-processor EmotionVAE checkpoint (517 MB) is required for
  evaluation and must be placed at
  `code/pretrained_models/post_processor/checkpoint.pth`. The PIRender
  renderer (234 MB) is needed only for FRRea rendering. Take both from the
  official baseline_react2026 repository.
- Python dependencies: `code/requirements.txt`.
- The MARS dataset is not included and must be obtained through the
  challenge organisers.