CAM-AH / README.md

Add CAM-AH ensemble-5 original weights (official Macro-F1 0.7456 @ thr 0.595)

2e0431f verified about 1 month ago

6.14 kB

tags:
  - audio
  - text
  - multimodal
  - cross-attention
  - multiple-instance-learning
  - ambivalence
  - hesitancy
  - abaw
  - pytorch-lightning

CAM-AH — Cross-Attention-Multimodal-Ambivalence-Hesitation

Original weights of the model behind our best official submission to the AH Video Recognition Challenge, 3rd edition (ABAW11 @ ECCV 2026): a 5-seed ensemble of an audio ⟷ text cross-attention network with psycholinguistic support features and gated attention-MIL pooling, run at a fixed decision threshold of 0.595.

Official private test (152 videos)	Value
Macro-F1	0.7456
Average precision (class 1)	0.8182
Decision threshold	0.595 (60 predicted positives)

Files

members/<run>/best-epoch=XX-val_ap=0.XXXX.ckpt   # original PyTorch Lightning checkpoints (5 members)
members/<run>/trainer_state.json                 # per-member dims / fusion options
ensemble_config.json                             # members, combination rule, threshold, hyperparameters

Each checkpoint is the unmodified Lightning file (weights + optimizer/loop state). The model weights live in state_dict under the model. prefix, about 2.96M parameters per member. The threshold field of each trainer_state.json is a per-member calibration. The ensemble ignores it and uses the fixed 0.595 from ensemble_config.json.

Model

Each video is sliced into 5 s windows (hop 2.5 s) aligned to the Whisper transcript by timestamp. Each window gets three feature vectors:

Input	Dim	Source
`feat_a` audio	320	librosa prosodic features (80 features × 4 statistics)
`feat_b` text	768	`cardiffnlp/twitter-roberta-base-emotion`, masked mean pool + L2
`feat_tab` support	74	question type (7) + acoustic hesitation (18) + textual A/H cues (49)

Per member, with inputs of shape (B, T, ·) and a padding mask (B, T):

Project audio and text to 512-d. Audio is the query and text is the key/value in an 8-head cross-attention, followed by a residual connection and LayerNorm.
Token fusion. The support features go through a masked BatchNorm, then Linear + ReLU + LayerNorm, and are concatenated with each window token before a fusion layer (512-d).
Gated attention-MIL pooling (Ilse et al., 2018) over the windows produces one vector per video.
An MLP head (512 → 512 → 1, dropout 0.3) outputs one logit per video.

Ensemble: average the 5 members' sigmoid probabilities. A video is predicted positive (1) when the mean probability is at least 0.595.

Training

Data: BAH train + val splits (902 videos). Checkpoint selection and early stopping used AP on the internal test split (525 videos); this is the val_ap in the file names.
Members: 5 identical models with seeds {42, 1, 2, 3, 4}.
Optimization: AdamW, lr 3e-4, weight decay 0.05, gradient clip 1.0, batch size 32, early stopping patience 20. The models saturate in 1–2 epochs.
Threshold: the ranking comes from the ensemble, and the operating point was fixed at 0.595 for the official test set. Its positive rate is lower than that of the internal calibration split, and the original calibrated threshold was 0.45 (official Macro-F1 0.6837).

Usage

The features have to be produced by the project's pipeline (make data) or by the self-contained notebook notebooks/ensemble5_pipeline.ipynb in the code repository. The network definition is src/models/cross_attention.py (_CrossAttentionFusionModule).

import json, torch
from huggingface_hub import snapshot_download

path = snapshot_download("LF-BF/CAM-AH")
cfg = json.load(open(f"{path}/ensemble_config.json"))

def load_member(ckpt_file, module):
    sd = torch.load(f"{path}/{ckpt_file}", map_location="cpu", weights_only=False)["state_dict"]
    module.load_state_dict({k.removeprefix("model."): v for k, v in sd.items()})
    return module.eval()

# members = [load_member(f, build_module(cfg["architecture"])) for f in cfg["members"]]
# with torch.no_grad():
#     p = torch.stack([m(feat_a, feat_b, pad_mask, feat_tab).sigmoid() for m in members]).mean(0)
# pred = (p >= cfg["threshold"]).long()

Data and terms of use

The model was trained on the BAH dataset (Behavioural Ambivalence/Hesitancy). The dataset is not included here and is distributed only by its authors under their EULA (github.com/sbelharbi/bah-dataset). These weights are intended for non-commercial academic research.

This work was carried out using the BAH dataset: a Multimodal Dataset – created and managed by the Montréal Behavioural Medicine Centre (MBMC: https://mbmc-cmcm.ca) and the Imaging, Vision and Artificial Intelligence Laboratory (LIVIA: https://www.etsmtl.ca/laboratoires/livia).

Citation

@article{martins2026audiotext,
  title   = {Audio-Text Cross-Attention with Psycholinguistic Support Features
             for Ambivalence/Hesitancy Recognition},
  author  = {Martins, Luiz F. B. F. and Pisaia, Rodrigo W. and Girardi, Matheus M.
             and Berkembrock, Isabella and Almeida, Jo{\~a}o A. and Hochuli, Andr{\'e} G.
             and Laroca, Rayson and Britto Jr., Alceu S.},
  journal = {arXiv preprint arXiv:2607.13345},
  year    = {2026},
  doi     = {10.48550/arXiv.2607.13345},
  url     = {https://arxiv.org/abs/2607.13345}
}

@inproceedings{gonzalez-26-bah,
  title     = {{BAH} Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change},
  author    = {González-González, M. and Belharbi, S. and Zeeshan, M. O. and Sharafi, M. and
               Aslam, M. H and Pedersoli, M. and Koerich, A. L. and Bacon, S. L. and Granger, E.},
  booktitle = {ICLR},
  year      = {2026}
}