---
tags:
  - audio
  - text
  - multimodal
  - cross-attention
  - multiple-instance-learning
  - ambivalence
  - hesitancy
  - abaw
  - pytorch-lightning
---

# CAM-AH — Cross-Attention-Multimodal-Ambivalence-Hesitation

Original weights of the model behind our **best official submission** to the
*AH Video Recognition Challenge, 3rd edition (ABAW11 @ ECCV 2026)*: a **5-seed ensemble** of an
audio ⟷ text cross-attention network with psycholinguistic support features and gated
attention-MIL pooling, run at a **fixed decision threshold of 0.595**.

| Official private test (152 videos) | Value |
|---|---|
| Macro-F1 | **0.7456** |
| Average precision (class 1) | 0.8182 |
| Decision threshold | 0.595 (60 predicted positives) |

- Paper: [*Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition*](https://arxiv.org/abs/2607.13345)
- Code: [Liga-de-IA-PUCPR/abaw-11-ah-challenge](https://github.com/Liga-de-IA-PUCPR/abaw-11-ah-challenge)

## Files

```
members/<run>/best-epoch=XX-val_ap=0.XXXX.ckpt   # original PyTorch Lightning checkpoints (5 members)
members/<run>/trainer_state.json                 # per-member dims / fusion options
ensemble_config.json                             # members, combination rule, threshold, hyperparameters
```

Each checkpoint is the unmodified Lightning file (weights + optimizer/loop state). The model
weights live in `state_dict` under the `model.` prefix, about 2.96M parameters per member.
The `threshold` field of each `trainer_state.json` is a per-member calibration. The ensemble
ignores it and uses the fixed 0.595 from `ensemble_config.json`.

## Model

Each video is sliced into **5 s windows (hop 2.5 s)** aligned to the Whisper transcript by
timestamp. Each window gets three feature vectors:

| Input | Dim | Source |
|---|---|---|
| `feat_a` audio | 320 | librosa prosodic features (80 features × 4 statistics) |
| `feat_b` text | 768 | [`cardiffnlp/twitter-roberta-base-emotion`](https://huggingface.co/cardiffnlp/twitter-roberta-base-emotion), masked mean pool + L2 |
| `feat_tab` support | 74 | question type (7) + acoustic hesitation (18) + textual A/H cues (49) |

Per member, with inputs of shape `(B, T, ·)` and a padding mask `(B, T)`:

1. Project audio and text to 512-d. Audio is the query and text is the key/value in an 8-head
   cross-attention, followed by a residual connection and LayerNorm.
2. **Token fusion.** The support features go through a masked BatchNorm, then Linear + ReLU +
   LayerNorm, and are concatenated with each window token before a fusion layer (512-d).
3. **Gated attention-MIL pooling** (Ilse et al., 2018) over the windows produces one vector
   per video.
4. An MLP head (512 → 512 → 1, dropout 0.3) outputs one logit per video.

**Ensemble:** average the 5 members' sigmoid probabilities. A video is predicted positive
(`1`) when the mean probability is at least **0.595**.

## Training

- Data: BAH `train` + `val` splits (902 videos). Checkpoint selection and early stopping
  used AP on the internal `test` split (525 videos); this is the `val_ap` in the file names.
- Members: 5 identical models with seeds `{42, 1, 2, 3, 4}`.
- Optimization: AdamW, lr 3e-4, weight decay 0.05, gradient clip 1.0, batch size 32, early
  stopping patience 20. The models saturate in 1–2 epochs.
- Threshold: the ranking comes from the ensemble, and the operating point was fixed at 0.595
  for the official test set. Its positive rate is lower than that of the internal
  calibration split, and the original calibrated threshold was 0.45 (official Macro-F1 0.6837).

## Usage

The features have to be produced by the project's pipeline (`make data`) or by the
self-contained notebook `notebooks/ensemble5_pipeline.ipynb` in the code repository. The
network definition is `src/models/cross_attention.py` (`_CrossAttentionFusionModule`).

```python
import json, torch
from huggingface_hub import snapshot_download

path = snapshot_download("LF-BF/CAM-AH")
cfg = json.load(open(f"{path}/ensemble_config.json"))

def load_member(ckpt_file, module):
    sd = torch.load(f"{path}/{ckpt_file}", map_location="cpu", weights_only=False)["state_dict"]
    module.load_state_dict({k.removeprefix("model."): v for k, v in sd.items()})
    return module.eval()

# members = [load_member(f, build_module(cfg["architecture"])) for f in cfg["members"]]
# with torch.no_grad():
#     p = torch.stack([m(feat_a, feat_b, pad_mask, feat_tab).sigmoid() for m in members]).mean(0)
# pred = (p >= cfg["threshold"]).long()
```

## Data and terms of use

The model was trained on the **BAH dataset** (Behavioural Ambivalence/Hesitancy). The
dataset is **not** included here and is distributed only by its authors under their EULA
([github.com/sbelharbi/bah-dataset](https://github.com/sbelharbi/bah-dataset)). These
weights are intended for non-commercial academic research.

This work was carried out using the BAH dataset: a Multimodal Dataset – created and managed by
the Montréal Behavioural Medicine Centre (MBMC: https://mbmc-cmcm.ca) and the Imaging, Vision
and Artificial Intelligence Laboratory (LIVIA: https://www.etsmtl.ca/laboratoires/livia).

## Citation

```bibtex
@article{martins2026audiotext,
  title   = {Audio-Text Cross-Attention with Psycholinguistic Support Features
             for Ambivalence/Hesitancy Recognition},
  author  = {Martins, Luiz F. B. F. and Pisaia, Rodrigo W. and Girardi, Matheus M.
             and Berkembrock, Isabella and Almeida, Jo{\~a}o A. and Hochuli, Andr{\'e} G.
             and Laroca, Rayson and Britto Jr., Alceu S.},
  journal = {arXiv preprint arXiv:2607.13345},
  year    = {2026},
  doi     = {10.48550/arXiv.2607.13345},
  url     = {https://arxiv.org/abs/2607.13345}
}

@inproceedings{gonzalez-26-bah,
  title     = {{BAH} Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change},
  author    = {González-González, M. and Belharbi, S. and Zeeshan, M. O. and Sharafi, M. and
               Aslam, M. H and Pedersoli, M. and Koerich, A. L. and Bacon, S. L. and Granger, E.},
  booktitle = {ICLR},
  year      = {2026}
}
```